Multisensory/Multimodal Interfaces for Robotic Surgery

The key innovation in this effort is the development of an open platform for augmented reality and haptic interfaces for robotic surgery. Intelligent Automation, Inc. (IAI) and Johns Hopkins University (JHU) propose to develop the software framework and surgeon interface to interface with any surgical robot. Key features of the surgeon interface include haptic manipulators, an autostereoscopic (3D) display, and augmented visual feedback using 3D overlays. The software framework will consist of four elements: control and command software, robot plug-ins, image processing plug-ins, and 3D reconstructions. The interface will also provide assisted autonomy through the use of adaptive virtual fixtures. Our team has extensive experience in robotic surgery, software design, haptics, and augmented reality. Dr. Okamurafs group has developed haptic and augmented reality interfaces for robotic surgery, and Dr. Rosner, our surgical consultant, has several years of experience using surgical robots. We propose to develop an open platform for augmented reality and haptic interfaces for robotic surgery in such a way that improves surgeon performance and returns to the surgeon a kinesthetic and somatosensory environment familiar to open procedures.