Reinforcement Learning for Assembly with Structured Graph Representations and Search

Abstract

Assembly problems are demanding as they require abstract high-level reasoning over action sequences together with a smooth execution of the corresponding low-level policy. In particular, learning to autonomously assemble complex 3D structures remains a challenging problem that includes decision making based on the target design and availability of building blocks with constraints regarding structural stability and robotic feasibility. To address the combinatorial complexity of the assembly tasks, we propose a multi-head attention graph representation that can be trained with reinforcement learning (RL) to encode the spatial relations and provide meaningful assembly actions. Combining structured representations with model-free RL and Monte-Carlo planning allows agents to operate with various target shapes and building block types. We design a hierarchical control framework that learns to sequence the building blocks to construct arbitrary 3D designs and ensures their feasibility, as we plan the geometric execution with the robot-in-the-loop. We demonstrate the flexibility of the proposed structured representation and our algorithmic solution in a series of simulated 3D assembly tasks with robotic evaluation, which showcases our method’s ability to learn to construct stable structures with a large number of building blocks.