GUARDIAN: Grounding visual data with actions for task verification

To support NASA missions that expand to the Moon, Mars and deeper into space, robots must be capable of increased levels of autonomous operation, such that they can provide effective support on ground and launch activities where human presence is either not viable, such as in hazardous environments, or in remote locations where human interaction is expected to be through remote, limited supervision. In this proposal, we focus on improving the state estimation capabilities of a robot system, such that during the execution of a long-horizon task, the robot agent will be capable of: (1) Descriptively assess its current state, by grounding visual data into predicates; and (2) Use this discrete representation to execute robot plans with closed-loop monitoring, where the visual feedback state is compared to the expected planned state, and if an anomaly is detected, the system is notified so appropriate action is taken.To provide robotic systems with the ability to ground visual feedback into symbolic states for long horizon robot planning to support sustained operation in space, TRACLabs proposes to develop GUARDIAN, a framework that allows a robot system to analyze RGB images, and infer predicates that describe its state, enabling it to perform close-loop task verification. GUARDIAN will be a system that bridges visual feedback information and transforms it into a symbolic state that will allow a robot to plan complex, multi-step tasks. For this, GUARDIAN will be implemented as a representation network that receives as input an image, and - after being trained in synthetic data labeled with efficient, weak supervision - outputs a set of predicates that describe the current state of the system, which can then be compared with the expected state, and thus anomalous situations can be timely detected.