Building An Agentic Physical AI System with reBot B601 and VLM Skills using Cosmos3-Edge

The project was developed by Seeed team Wenhan Fan, Yaohui Zhu, Youjiang Yu, Junjie Liu, Vector Li, Elaine Wu; together with the NVIDIA team Asier Arranz, Johnny Núñez Cano, Minto Zheng, and  Abhilash Somasamudramath.

From Natural-Language Commands to Autonomous Perception, Planning, Action, and Recovery

Live at WRC 2026 (Booth A435), Seeed is taking physical AI a step further with the reBot B601 RS. Instead of treating foundation models as simple function callers, we deploy the VLM as an agentic brain calling 21 native Action Skills, enabling real-time closed-loop perception, execution, and dynamic recovery with a safety harness.

Traditional robot arms execute predefined programs or trained policies. Even many VLM-powered robot demos still follow a relatively simple pattern: understand a command, call a predefined function, and execute it.

With reBot B601 RS × Rebot Dev Arm and Action Skills, we are taking a step further.

Given a high-level goal, such as “Put the red block on the blue box”, the robot independently manages the task via a continuous closed loop:

Perceive → Reason → Plan → Select Skills → Act → Observe → Recover → Re-Plan → Learn

---

Core System Architecture

Powered by NVIDIA Cosmos™ & Cosmos 3-Edge. Built on NVIDIA Cosmos™, a world foundation model platform designed to accelerate Physical AI development by enabling machines to understand, simulate, and interact with the physical world Cosmos 3-Edge serves as the on-device reasoning backbone. By ingesting multimodal inputs, including text, images, video streams, and past action trajectories, Cosmos 3-Edge generates coherent multi-step reasoning, future-state predictions, and actionable outputs tailored for real-world physical manipulation. Then the system saves spatial and temporal information and learns from failures, so it is like “agentic reinforcement learning”. 

1. Physical Action Space (Native Skills)

Physical capabilities are encapsulated into 21 structured Tool-Use interfaces (e.g., grasp, place, hand over, count, emergency stop) rather than low-level joint offsets.

• Dynamic Composition: The VLM dynamically selects and chains skills based on visual feedback.
• Closed-Loop Recovery: If an action fails (e.g., verify_grasp() returns false), the agent reassesses the scene, modifies parameters, and retries another strategy rather than failing blindly.

2. Three-Tier Hierarchical Control

Physical robotics requires decision-making across different time scales. reBot separates latency demands into three distinct layers:

• Reflex Layer (500 Hz): Runs directly on hardware, managing millisecond-level motor control, kinematic constraints, and safety triggers.
• Habit Layer: Executes cached, high-frequency routines quickly without calling the VLM.
• VLM Agent Layer: Manages high-level semantic reasoning, multi-step planning, and global failure handling.

3. State Memory & Deterministic Safety

• Belief Store: Combines ~15 seconds of short-term scene memory with persistent object tracking, preserving state context even during temporary visual occlusions.
• Safety Harness: Maintains a strict boundary between reasoning and motor execution. The VLM decides intent, but every skill must pass a deterministic safety harness before reaching the 500 Hz real-time controller.
• Agentic “Reinforcement Learning”: Agents can learn from failures, creating new skills and documenting everything, so they can learn from previous actions for future similar actions, improving success rate over time. 

From Scripted Arms to Agentic Endpoints

The true value of multimodal models in robotics is handling real-world edge cases. By coupling high-level VLM reasoning with a deterministic 500 Hz safety envelope, reBot B601 RS transforms from a pre-programmed arm into an adaptive endpoint for physical AI developers.

Get Started with reBot & VLM Skills

The combination of open-source hardware, deterministic real-time control, and high-level reasoning opens up new possibilities for physical AI research. We are rolling out the reBot B601 RS × VLM Skills toolkit to help developers bridge the gap between foundation models and physical actuation.

• Visit Booth A435 at WRC 2026: See a live interactive demo in person.
• Build Custom Skills: Add your own vision models or motion pipelines to expand the robot’s action space.
• Join the Community: Share your agentic workflows and testbed setups with fellow physical AI developers.

About Author