SUN Programs: Unifying Control and Learning in Long-Horizon Manipulation

SUN Programs: Unifying Control and Learning in Long-Horizon Manipulation

Introduction

In the realm of artificial intelligence, there has been a longstanding debate between model-based control and learned policies, particularly in long-horizon manipulation tasks. Model-based control excels at executing specified objectives with precision, while learned policies can generalize behavior through reinforcement learning (RL). However, existing approaches often discard task semantics, relying on hand-crafted rewards that can drift from the intended goals.

To address this issue, researchers have introduced Semantically UNified (SUN) Programs. These are typed executables designed to bridge model-based control and learned policies in a way that retains task semantics throughout training. This article delves into the concept of SUN Programs and their implementation through Kuafu, an automated synthesis system.

The Problem with Existing Approaches

In current practice, control systems execute predefined tasks efficiently but may not adapt well to varying conditions or changes in objectives. On the other hand, learned policies can reactively handle a wide range of situations but often lack the robustness and precision that control offers. This discrepancy has led to a situation where task semantics are frequently lost during the transition from model-based control to RL.

Enter SUN Programs

SUN Programs aim to unify these two approaches by defining geometric and contact relations once, which can then be compiled into various components of Model Predictive Control (MPC). These components include MPC costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. By doing so, SUN Programs ensure that task semantics are preserved throughout the training process.

Kuafu: Synthesizing SUN Programs from Language and Scene Semantics

The Kuafu system leverages large vision language models to automatically synthesize SUN Programs based on both language descriptions and scene semantics. It screens feasibility through MPC simulations, ensuring that generated policies can be reliably executed in real-world scenarios.

Performance Evaluation

Across nine different manipulation tasks, Kuafu demonstrated superior performance compared to existing baselines such as sparse-reward methods (35.67%) and Stage-BC (24.75%). Specifically, Kuafu achieved a macro-success rate of 82.03%. Additionally, at an 8192-way scale, it generated 10.57 times more successful trajectories per hour than human teleoperation.

Training Stage-Conditioned Policies

Using Kuafu-generated SUN Programs, researchers were able to train Deep Deterministic Policy Gradients (DP3) policies that showed significantly better performance in simulation and on physical robots. With just 500 trajectories per task, DP3 achieved a simulation success rate of 46.0% compared to alternatives at 22.4%, and on physical Franka and Kinova robots, it reached a success rate of 34.7%.

Conclusion

These results highlight the effectiveness of using simulation-screened task semantics in training robust policies without the need for demonstrations or manually crafted dense rewards. SUN Programs represent a significant step forward in unifying symbolic planning with data-driven execution, offering a promising solution to the challenges faced by current AI systems in long-horizon manipulation tasks.

Keywords

• Recherche en intelligence artificielle
• Model Predictive Control (MPC)
• Semantically UNified (SUN) Programs
• Kuafu system
• Deep Deterministic Policy Gradients (DP3)