Quick Start

Get started with LeRobot in minutes

Overview

This guide will walk you through the essential workflows of LeRobot:

1. Loading and exploring datasets
2. Training a policy
3. Evaluating a trained policy
4. Running inference on real hardware

Make sure you have installed LeRobot before proceeding.

Working with Datasets

Loading a Dataset

LeRobot datasets are hosted on Hugging Face Hub and can be loaded with a single line:

from lerobot.datasets.lerobot_dataset import LeRobotDataset

# Load a dataset from the Hub
dataset = LeRobotDataset("lerobot/aloha_mobile_cabinet")

print(f"Number of episodes: {dataset.num_episodes}")
print(f"Number of frames: {dataset.num_frames}")

Exploring Dataset Metadata

Before downloading the full dataset, you can explore its metadata:

from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata

# Load only metadata (lightweight)
ds_meta = LeRobotDatasetMetadata("lerobot/aloha_mobile_cabinet")

print(f"Total episodes: {ds_meta.total_episodes}")
print(f"Frames per second: {ds_meta.fps}")
print(f"Robot type: {ds_meta.robot_type}")
print(f"Camera keys: {ds_meta.camera_keys}")
print(f"Tasks: {ds_meta.tasks}")

Accessing Dataset Items

# Access a single frame
frame = dataset[0]
print(f"Action shape: {frame['action'].shape}")
print(f"State shape: {frame['observation.state'].shape}")

# Access camera images (if available)
if ds_meta.camera_keys:
    camera_key = ds_meta.camera_keys[0]
    print(f"Image shape: {frame[camera_key].shape}")  # (C, H, W) in PyTorch format

Loading Temporal Data

For robot learning, you often need sequences of past observations and future actions:

# Define temporal offsets
delta_timestamps = {
    # Load 4 images: 1s before, 0.5s before, 0.2s before, and current
    "observation.image": [-1.0, -0.5, -0.2, 0.0],
    # Load 6 state vectors
    "observation.state": [-1.5, -1.0, -0.5, -0.2, -0.1, 0.0],
    # Load 64 future actions for action chunking
    "action": [t / ds_meta.fps for t in range(64)],
}

dataset = LeRobotDataset(
    "lerobot/aloha_mobile_cabinet",
    delta_timestamps=delta_timestamps
)

frame = dataset[0]
print(f"Image temporal shape: {frame['observation.image'].shape}")  # (4, C, H, W)
print(f"State temporal shape: {frame['observation.state'].shape}")  # (6, state_dim)
print(f"Action temporal shape: {frame['action'].shape}")  # (64, action_dim)

Using with PyTorch DataLoader

import torch

dataloader = torch.utils.data.DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4,
    pin_memory=True,
)

for batch in dataloader:
    print(f"Batch action shape: {batch['action'].shape}")  # (32, 64, action_dim)
    break

Streaming Large Datasets

For very large datasets, use streaming mode to avoid downloading everything:

from lerobot.datasets.streaming_dataset import StreamingLeRobotDataset

# Stream the massive DROID dataset (26M frames, 4TB!)
dataset = StreamingLeRobotDataset(
    "lerobot/droid_1.0.1",
    delta_timestamps={"action": [0.0]},
    tolerance_s=1e-3
)

# Data is fetched on-demand during iteration
dataloader = torch.utils.data.DataLoader(dataset, batch_size=16, num_workers=4)

Training Your First Policy

Using the Command-Line Interface

The simplest way to train a policy:

lerobot-train \
  --policy=act \
  --dataset.repo_id=lerobot/aloha_mobile_cabinet

This command:

• Downloads the dataset
• Configures the ACT policy with default settings
• Trains the model with automatic logging to Weights & Biases
• Saves checkpoints periodically

Run lerobot-train --help to see all available options and configurations.

Training Programmatically

For more control, train policies programmatically:

1. Import Dependencies

   from pathlib import Path
   import torch
   from lerobot.configs.types import FeatureType
   from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
   from lerobot.datasets.utils import dataset_to_policy_features
   from lerobot.policies.act.configuration_act import ACTConfig
   from lerobot.policies.act.modeling_act import ACTPolicy
   from lerobot.policies.factory import make_pre_post_processors
   
2. Configure Training

   # Setup
   output_dir = Path("outputs/train/my_act_policy")
   output_dir.mkdir(parents=True, exist_ok=True)
   device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
   
   # Dataset configuration
   dataset_id = "lerobot/aloha_mobile_cabinet"
   dataset_metadata = LeRobotDatasetMetadata(dataset_id)
   
3. Create Policy

   # Extract features for policy configuration
   features = dataset_to_policy_features(dataset_metadata.features)
   output_features = {key: ft for key, ft in features.items() if ft.type is FeatureType.ACTION}
   input_features = {key: ft for key, ft in features.items() if key not in output_features}
   
   # Initialize policy with configuration
   cfg = ACTConfig(input_features=input_features, output_features=output_features)
   policy = ACTPolicy(cfg)
   policy.train()
   policy.to(device)
   
   # Create preprocessor and postprocessor
   preprocessor, postprocessor = make_pre_post_processors(
       cfg, dataset_stats=dataset_metadata.stats
   )
   
4. Prepare Dataset

   # Configure temporal data loading for ACT
   delta_timestamps = {
       "action": [i / dataset_metadata.fps for i in cfg.action_delta_indices],
   }
   
   # Add image features if present
   delta_timestamps.update({
       k: [i / dataset_metadata.fps for i in cfg.observation_delta_indices]
       for k in cfg.image_features
   })
   
   # Load dataset
   dataset = LeRobotDataset(dataset_id, delta_timestamps=delta_timestamps)
   
   # Create dataloader
   optimizer = torch.optim.Adam(policy.parameters(), lr=1e-4)
   dataloader = torch.utils.data.DataLoader(
       dataset,
       batch_size=32,
       shuffle=True,
       num_workers=4,
       pin_memory=device.type != "cpu",
       drop_last=True,
   )
   
5. Training Loop

   # Training configuration
   training_steps = 5000
   log_freq = 100
   
   # Training loop
   step = 0
   done = False
   while not done:
       for batch in dataloader:
           # Preprocess batch
           batch = preprocessor(batch)
           
           # Forward pass
           loss, _ = policy.forward(batch)
           
           # Backward pass
           loss.backward()
           optimizer.step()
           optimizer.zero_grad()
           
           # Logging
           if step % log_freq == 0:
               print(f"Step {step}: loss = {loss.item():.4f}")
           
           step += 1
           if step >= training_steps:
               done = True
               break
   
6. Save Model

   # Save checkpoint locally
   policy.save_pretrained(output_dir)
   preprocessor.save_pretrained(output_dir)
   postprocessor.save_pretrained(output_dir)
   
   # Optionally push to Hub
   # policy.push_to_hub("<your-username>/my_act_policy")
   # preprocessor.push_to_hub("<your-username>/my_act_policy")
   # postprocessor.push_to_hub("<your-username>/my_act_policy")
   

The complete training example is available in the repository at examples/training/train_policy.py.

Evaluating Policies

Evaluation in Simulation

Evaluate your trained policy in a simulation environment:

lerobot-eval \
  --policy.path=lerobot/pi0_libero_finetuned \
  --env.type=libero \
  --env.task=libero_object \
  --eval.n_episodes=10

Supported simulation environments:

• ALOHA: Bimanual manipulation tasks
• PushT: 2D pushing task
• LIBERO: Benchmark for lifelong robot learning (Linux only)
• MetaWorld: Meta-learning benchmark with 50 tasks

Evaluation Results

The evaluation script outputs:

• Success rate across episodes
• Average reward per episode
• Episode lengths
• Videos of policy rollouts (saved to output directory)

Running on Real Hardware

Basic Inference Loop

Deploy a trained policy on real hardware:

import torch
from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.utils import build_inference_frame, make_robot_action
from lerobot.robots.so_follower import SO100Follower, SO100FollowerConfig
from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig

# Load trained policy
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_id = "<your-username>/my_act_policy"
policy = ACTPolicy.from_pretrained(model_id)
policy.to(device)
policy.eval()

# Load dataset metadata for preprocessing
dataset_id = "lerobot/aloha_mobile_cabinet"
dataset_metadata = LeRobotDatasetMetadata(dataset_id)
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, dataset_stats=dataset_metadata.stats
)

# Configure robot and cameras
camera_config = {
    "side": OpenCVCameraConfig(index_or_path=0, width=640, height=480, fps=30),
    "top": OpenCVCameraConfig(index_or_path=1, width=640, height=480, fps=30),
}

robot_cfg = SO100FollowerConfig(
    port="/dev/ttyUSB0",
    id="my_robot",
    cameras=camera_config
)

robot = SO100Follower(robot_cfg)
robot.connect()

# Run inference loop
max_steps = 100
for step in range(max_steps):
    # Get observation from robot
    obs = robot.get_observation()
    
    # Build frame matching dataset format
    obs_frame = build_inference_frame(
        observation=obs,
        ds_features=dataset_metadata.features,
        device=device
    )
    
    # Preprocess observation
    obs_frame = preprocessor(obs_frame)
    
    # Get action from policy
    with torch.no_grad():
        action = policy.select_action(obs_frame)
    
    # Postprocess action
    action = postprocessor(action)
    
    # Convert to robot action format
    action = make_robot_action(action, dataset_metadata.features)
    
    # Send action to robot
    robot.send_action(action)

robot.disconnect()

Always test your policy in simulation before deploying on real hardware. Start with slow speeds and ensure emergency stops are accessible.

Hardware-Specific Setup

For hardware-specific configuration:

# Find camera indices
lerobot-find-cameras

# Find robot serial port
lerobot-find-port

# Calibrate robot motors
lerobot-calibrate --robot=so100

# Set up motors (if needed)
lerobot-setup-motors --robot=so100 --port=/dev/ttyUSB0

Data Collection

Collect your own demonstrations:

1. Set Up Teleoperation

   # Teleoperate the robot to test control
   lerobot-teleoperate \
     --robot=so100 \
     --robot.port=/dev/ttyUSB0 \
     --teleoperator=gamepad
   
2. Record Demonstrations

   # Record demonstrations to a new dataset
   lerobot-record \
     --robot=so100 \
     --robot.port=/dev/ttyUSB0 \
     --teleoperator=gamepad \
     --output_dir=data/my_task \
     --num_episodes=50
   
3. Replay and Verify

   # Replay recorded episodes to verify quality
   lerobot-replay \
     --dataset_dir=data/my_task \
     --episode_index=0
   
4. Push to Hub (Optional)

   from lerobot.datasets.lerobot_dataset import LeRobotDataset
   
   # Load local dataset
   dataset = LeRobotDataset.from_disk("data/my_task")
   
   # Push to Hugging Face Hub
   dataset.push_to_hub("<your-username>/my_task")
   

Visualizing Datasets

Visualize datasets to understand their structure:

# Launch interactive visualization
lerobot-dataset-viz --dataset.repo_id=lerobot/aloha_mobile_cabinet

This opens a Rerun viewer showing:

• Camera feeds synchronized across time
• Robot state trajectories
• Action commands
• Episode boundaries

Example Workflows

Workflow 1: Train on Existing Dataset

CLI

# Train ACT on ALOHA dataset
lerobot-train \
  --policy=act \
  --dataset.repo_id=lerobot/aloha_mobile_cabinet \
  --training.num_epochs=100 \
  --training.batch_size=32

# Evaluate in simulation
lerobot-eval \
  --policy.path=outputs/train/act \
  --env.type=aloha \
  --eval.n_episodes=10

from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
from lerobot.policies.act.configuration_act import ACTConfig
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.datasets.utils import dataset_to_policy_features
from lerobot.configs.types import FeatureType
import torch

# Load and configure
dataset_id = "lerobot/aloha_mobile_cabinet"
metadata = LeRobotDatasetMetadata(dataset_id)
features = dataset_to_policy_features(metadata.features)

output_features = {k: ft for k, ft in features.items() if ft.type is FeatureType.ACTION}
input_features = {k: ft for k, ft in features.items() if k not in output_features}

# Create policy
cfg = ACTConfig(input_features=input_features, output_features=output_features)
policy = ACTPolicy(cfg)
preprocessor, postprocessor = make_pre_post_processors(cfg, dataset_stats=metadata.stats)

# Train...

Workflow 2: Collect Data and Train

# 1. Collect demonstrations
lerobot-record --robot=so100 --output_dir=data/my_task --num_episodes=50

# 2. Visualize collected data
lerobot-dataset-viz --dataset_dir=data/my_task

# 3. Train policy on collected data
lerobot-train --policy=act --dataset.local_dir=data/my_task

# 4. Deploy on robot
# (Use Python script for hardware inference)

Workflow 3: Fine-tune Pretrained Model

# Fine-tune a pretrained policy
lerobot-train \
  --policy.pretrained_path=lerobot/pi0_fast \
  --dataset.repo_id=<your-username>/my_custom_data \
  --training.learning_rate=1e-5 \
  --training.num_epochs=20

Next Steps

Hardware Integration

Set up your robot hardware and cameras

Policy Documentation

Learn about available policies and their configurations

Dataset Tools

Advanced dataset manipulation and tools

Examples

Explore more examples in the GitHub repository

Common Patterns

Pattern: Load, Train, Save

import torch
from lerobot.datasets.lerobot_dataset import LeRobotDataset, LeRobotDatasetMetadata
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.act.configuration_act import ACTConfig
from lerobot.policies.factory import make_pre_post_processors
from lerobot.datasets.utils import dataset_to_policy_features
from lerobot.configs.types import FeatureType

# Load dataset and metadata
dataset_id = "lerobot/pusht"
metadata = LeRobotDatasetMetadata(dataset_id)
features = dataset_to_policy_features(metadata.features)

# Configure policy
output_features = {k: ft for k, ft in features.items() if ft.type is FeatureType.ACTION}
input_features = {k: ft for k, ft in features.items() if k not in output_features}
cfg = ACTConfig(input_features=input_features, output_features=output_features)

# Initialize
policy = ACTPolicy(cfg)
preprocessor, postprocessor = make_pre_post_processors(cfg, dataset_stats=metadata.stats)

# Train (simplified)
optimizer = torch.optim.Adam(policy.parameters(), lr=1e-4)
# ... training loop ...

# Save
policy.save_pretrained("outputs/my_policy")
preprocessor.save_pretrained("outputs/my_policy")
postprocessor.save_pretrained("outputs/my_policy")

Pattern: Load and Infer

import torch
from lerobot.policies.act.modeling_act import ACTPolicy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.datasets.lerobot_dataset import LeRobotDatasetMetadata

# Load pretrained policy
policy = ACTPolicy.from_pretrained("<username>/my_policy")
policy.eval()

# Load preprocessor/postprocessor
dataset_id = "lerobot/aloha_mobile_cabinet"
metadata = LeRobotDatasetMetadata(dataset_id)
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, dataset_stats=metadata.stats
)

# Run inference
with torch.no_grad():
    obs = preprocessor(observation)
    action = policy.select_action(obs)
    action = postprocessor(action)

Additional Resources

• Tutorial Space: Free hands-on robot learning course
• Example Scripts: GitHub examples directory
• Discord: Join the community
• Documentation: Full API reference

Explore the examples/ directory in the repository for complete, runnable examples covering ACT, Diffusion Policy, VQ-BeT, and more.