Abstract
A general-purpose agent directly controls a physical robot by interpreting visuals, writing executable programs, and revising actions based on physical feedback across diverse manipulation tasks.
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
Community
Agent as Policy (AGP) connects a general purpose coding agent directly to a physical robot. The agent interprets camera observations, writes programs, requests motions, and adjusts its actions using physical feedback while model weights stay fixed. Evaluated tasks include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves at least 8 successful trials out of 10 for each evaluated configuration in assembly, block construction, and dice flipping. Reusing saved procedures and programs reduces execution time across repeated trials.
A short video showing AGP on real robot tasks and how the agent controls the robot through tool calls and visual feedback.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- ETA: A New Agentic Paradigm for Embodied Tasks (2026)
- HODAgent: Towards On-Demand, Responsive Humanoids for Physical World Human Interaction (2026)
- Practice Makes Policies: Bootstrapping and Consolidating Robotic Capabilities from Zero Human Demonstrations (2026)
- A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution (2026)
- Teach and Grow: An Agent-Centered Architecture for General Robot Learning (2026)
- 2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation (2026)
- Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2609.12541 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper