SBIR Phase II: Intelligent Robotic Foliage Manipulation for Faster Autonomous Harvesting

The broader/commercial impact of this Small Business Innovation Research (SBIR) Phase II project will be in development of a dual-arm robot with advanced harvesting capabilities, a bottleneck that has stymied the development of earlier robotic harvesters.  By using multiple fixed and moving cameras, the robot's motion is planned and data obtained are used to co-optimize exploration and picking. This approach is generalizable and will avoid unnecessary motions and speed the operation significantly. This technology will provide more flexibility to manage demand spikes during the short peak season. The same technology can extend to other harvesting applications. The high-risk innovation in this project is an Interactive Perception framework that will let a dual-arm robotic harvester decide, in real time, how to interleave foliage-pushing and picking motions to keep throughput high in a partly hidden, constantly changing scene. The difficulty: every action alters the scene the robot is trying to perceive, and the space of possible action sequences grows exponentially with the number of objects and viewpoints. Earlier handcrafted, sequential perceive-then-act behaviors proved too slow and rigid to generalize. Solving this is the gating problem for commercially viable robotic harvesting. This project will advance autonomous harvesting from a working prototype to a commercially viable picking rate. Three challenges will be addressed: development of a parametric, real-time state-estimation method that fuses images from several fixed cameras and one moving end-effector camera at about ten frames per second; development of a motion-planning software for a fourteen-degree-of-freedom dual-arm system on linear rails, able to reach the full bed width including the shoulders; and development of an Interactive Perception decision layer that co-optimizes exploration and picking, so the arms stop wasting time on low-payoff motions. The work will combine simulation-based software development with repeated laboratory and field testing on a research prototype. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.