Handler: An End-to-End Tool for Semantic-based Manipulation using Affordance Templates

We propose to research, develop, and demonstrate Handler, an autonomous semantic detection, planning, and grasping affordance module capable of fast online inference and adaptation, as well as continuous learning and improvement. Handler will work by combining semantic and primitive pose-recognition algorithms with a rich affordance template library and online grasp-finding algorithms. Uniquely, it will also feature a pipeline for expanding its semantic recognition and affordance library through continuous learning using state-of-the-art mesh generation tools. It will then estimate grasping locations from learned grasp generation models using these created meshes.nbsp;Handler will consist of four primary tools:Handler Environment Constructor, whichnbsp;will leverage state-of-the-art object classifier and pose extraction algorithms to automatically create a digital twin of the real environment and the objects within.nbsp;Handler Dynamic Affordance Template Library, which is a database of modifiable objects that encode anbsp;mesh and defined object interactions, including candidate grasp locations and suggested trajectories between grasp points. These templates can be applied to objects found by the environment constructor, andnbsp;govern how they can be manipulated by robots.Handler User Interface, whichnbsp;allows users to view the digital twin of the environment and manage the affordance library, includingnbsp;tweaking existing affordance templates, adjusting upcoming interactions or grasp methods, or capturing perception and semantic data for construction of new templates.nbsp;Handler Affordance Builder, which can automatically create object meshes from captured video and use them to both train pose estimation and semantic classifier networks, as well as create new affordance templates using automatic grasp calculators.