Rad-Hard Adaptive Dual-Mode Event-Based Vision and Perception for Autonomous Robot Operations

In response to NASA STTR topic T4.01, Information Technologies for Intelligent and Adaptive Space Robotics, Alphacore Inc. in partnership with the Arizona State University (ASU) School of Earth and Space Exploration will develop a low-SWaP-C, high-performance, extreme perception and vision system for autonomous robot operations. Our novel radiation-hard adaptive dual-mode neuromorphic (event-based) vision system configuration is designed to provide terrestrial autonomous robot-comparable 3D object detection, depth estimation, mapping, and tracking functionality for future use on lunar and planetary surfaces.nbsp;Our approach is to recognize that ultra-high performance terrestrial state-of-the-art image processing microelectronics hardware likely will not be available. To address this constraint, our solution is to instead selectively reduce or throttle the image data from the image sensors to the downstream image processing microelectronic hardware. By selectively (and significantly) reducing the image data rate from the camera(s), lower performance space-qualifiable image processing electronics will then be able to provide the needed functionality on the now much sparser information from image data. The challenge is reducing the image data rate, while still providing the terrestrial-comparable state (pose and velocity) estimation, 3D object detection, depth estimation, mapping, and tracking functionality needed for autonomous operations.nbsp;We propose to solve this challenge by replacing existing conventional space-grade CMOS frame-based cameras with a novel radiation-hardened version of a Dynamic and Active Pixel Vision Sensor (DAVIS) dual-mode image sensor , which combines a conventional global-shutter CMOS camera with an event-based image sensor (EBS) in the same pixel array. In practice, event-based image sensors can reduce the downstream computational burden by an estimated one to two orders of magnitude.