ioailab

ioailab provides G1 robot cfgs, an IsaacLab-style task registry, action/sensor helpers, dataset refs, and action agents for IsaacLab.

make shell-gui
python examples/01_collect.py

Runtime boundary — import ioailab is side-effect free. Task IDs are registered only through explicit imports under ioailab.tasks; IsaacLab owns app launch, env construction, managers, sensors, PhysX, and stepping.

Rendered scenes

Stack cube

The loop

Every workflow is the same explicit loop: build an env, build an agent, step, and records data through env.collect(...).

from ioailab.agents import CuroboPlannerAgent
from ioailab.envs import make_env

task_id = "GalbotG1-PickCube-v0"
env = make_env(task_id, num_envs=1)
agent = CuroboPlannerAgent.from_task(task_id)

dataset = env.collect(
    agent=agent,
    episodes=1,
    path="data/pick_cube_demos.hdf5",
)

CuroboPlannerAgent, PolicyAgent, TeleopAgent, and TaskFlowAgent are interchangeable at this boundary. env.collect(...) and env.evaluate(...) remain convenience helpers for batch data generation and metrics.

Current surface

Where to go next

AGENTS.md and Architecture are the design source of truth.

Tutorial

Welcome to ioailab, the IOAI-oriented simulation platform built by the Galbot simulation team. This tutorial shows how to build a task step by step, starting from a simple PickCube task and ending with larger compound tasks that run through several phases.

The tutorial is split into three chapters. Read them in order: the first chapter teaches the basic task-generation loop, and the later chapters reuse that loop for component tasks and compound tasks.

simple task
-> component tasks
-> compound task

• Chapter 1: Build and Activate a Simple Task
• Chapter 2: Build PickToShelf From Component Tasks
• Chapter 3: Build SortToShelf With Task Options and Sequence Agents

The main workflow is iterative. A new task should become runnable before it becomes complete: first make IsaacLab construct the environment, then run one episode, refine the MDP and reset behavior, collect data, and only then add Mimic, training, evaluation, or compound-task structure. It is normal to return to an earlier step when a later step exposes a weak termination, observation, or initial-state definition.

Keep task MDP functions under semantic owners such as events.py, observations.py, rewards.py, and terminations.py. Do not add a generic terms.py bucket.

minimal task
-> single-env smoke
-> optional teleop
-> refine MDP/reset
-> multi-env smoke
-> collect data
-> optional mimic dataset expansion
-> train
-> evaluate
-> revisit task definition

Chapter 1: Build and Activate a Simple Task

Reference task: GalbotG1-PickCube-v0.

Run interactive examples from the GUI container:

make shell-gui

Workflow:

minimal task -> single-env smoke -> optional teleop -> refine MDP/reset
-> multi-env smoke -> collect data -> optional mimic -> train -> evaluate

1. Task Files

PickCube is the smallest reference layout:

src/ioailab/tasks/pick_cube/
  __init__.py
  scene.py
  mdp/
    events.py
    terminations.py
  config/g1/
    env_cfg.py
    mdp_cfg.py
    agent_cfg/motion_plan.py

Source map:

The task ID is the public interface:

GALBOT_G1_PICK_CUBE_TASK = TaskSpec(
    task_id="GalbotG1-PickCube-v0",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    isaaclab_kwargs={
        "env_cfg_entry_point": (
            "ioailab.tasks.pick_cube.config.g1.env_cfg:"
            "GalbotG1PickCubeEnvCfg"
        ),
    },
    motion_plan_entry_point=(
        "ioailab.tasks.pick_cube.config.g1.agent_cfg.motion_plan:"
        "PickCubeMotionPlan"
    ),
)

Verify construction:

python - <<'PY'
from ioailab.envs import make_env

env = make_env("GalbotG1-PickCube-v0", num_envs=1)
env.close()
PY

2. Single-Env Smoke

Source file: examples/01_collect.py.

Run one visible episode:

python examples/01_collect.py \
  --task GalbotG1-PickCube-v0 \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000

The default agent is resolved from the task:

task_id = args.task
agent = CuroboPlannerAgent.from_task(task_id)
env = make_env(task_id, num_envs=args.num_envs, headless=args.headless)

3. Optional Teleop

Source: examples/01_collect.py.

For GP001 demos, uncomment the teleop block in examples/01_collect.py:

# from ioailab.agents import TeleopAgent
# task_id = "GalbotG1-PickCube-Teleop-v0"
# agent = TeleopAgent.from_device("gp001", task=task_id)

Then run the same collector:

python examples/01_collect.py --episodes 1 --num-envs 1

4. Refine The Task

After the first visible run, refine only the task-owned pieces:

Do not add a generic terms.py bucket; keep terms in named files such as events.py and terminations.py.

Termination shape:

@configclass
class PickCubeTerminationsCfg:
    time_out = DoneTerm(func=base_mdp.time_out, time_out=True)
    released_on_blue_block = make_pick_cube_release_termination_term()

5. Multi-Env Smoke

Source files: examples/01_collect.py, src/ioailab/envs/env.py.

python examples/01_collect.py \
  --task GalbotG1-PickCube-v0 \
  --episodes 1 \
  --num-envs 4 \
  --max-steps 1000

This checks batched reset, observation shape, action shape, and per-row termination.

6. Collect Data

Source files: examples/01_collect.py, src/ioailab/envs/env.py.

python examples/01_collect.py \
  --task GalbotG1-PickCube-v0 \
  --episodes 10 \
  --num-envs 1 \
  --dataset-path data/pick_cube_demos.hdf5

To save the final scene as a scenario YAML:

Source files: examples/01_collect.py, src/ioailab/tasks/common/scenario.py.

python examples/01_collect.py \
  --task GalbotG1-PickCube-v0 \
  --episodes 1 \
  --num-envs 1 \
  --dataset-path data/pick_cube_scenario_source.hdf5 \
  --save-end-scenario data/pick_cube/end.yaml

7. Optional: use mimic to expand dataset

Source files: examples/02_mimic.py, src/ioailab/tasks/pick_cube/config/g1/env_cfg.py.

Mimic expands a dataset through DatasetRef.task_id; for PickCube that resolves to GalbotG1-PickCube-Mimic-v0.

python examples/02_mimic.py \
  --task GalbotG1-PickCube-v0 \
  --dataset-path data/pick_cube_demos.hdf5 \
  --output-path data/pick_cube_demos_mimic.hdf5 \
  --episodes 36 \
  --num-envs 9

8. Train

Source: examples/03_train.py.

python examples/03_train.py \
  --task GalbotG1-PickCube-v0 \
  --dataset-path data/pick_cube_demos_mimic.hdf5 \
  --output-dir outputs/pick_cube \
  --epochs 1

9. Evaluate

Source: examples/04_eval.py.

python examples/04_eval.py \
  --task GalbotG1-PickCube-v0 \
  --checkpoint outputs/pick_cube/model_best_training.pth \
  --episodes 36 \
  --num-envs 9 \
  --max-steps 1000

If evaluation exposes a bad reset, weak observation, or wrong termination, return to the task files first. Training changes should come after the task contract is stable.

Chapter 2: Build PickToShelf From Component Tasks

PickToShelf is built from three standalone tasks:

Pick -> Nav -> Place

Run each component first. Then run the coherent task.

1. Run Pick

Use examples/06_collect_component_task.py.

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/pick_to_shelf_pick/__init__.py.

Keep this preset active:

COMPONENT_PRESET = "pick_to_shelf_pick"

Run:

python examples/06_collect_component_task.py \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000 \
  --dataset-path data/pick_to_shelf_pick.hdf5

2. Run Nav

In examples/06_collect_component_task.py, switch one preset line:

# COMPONENT_PRESET = "pick_to_shelf_pick"
COMPONENT_PRESET = "pick_to_shelf_nav"

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/pick_to_shelf_nav/agent.py.

Run:

python examples/06_collect_component_task.py \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000 \
  --dataset-path data/pick_to_shelf_nav.hdf5

3. Run Place

Switch examples/06_collect_component_task.py to Place:

# COMPONENT_PRESET = "pick_to_shelf_nav"
COMPONENT_PRESET = "pick_to_shelf_place"

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/pick_to_shelf_place/__init__.py.

Run:

python examples/06_collect_component_task.py \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000 \
  --dataset-path data/pick_to_shelf_place.hdf5

4. Scenario Starts

Scenario YAML is for standalone component starts. It is not used inside the coherent task.

Save a nav-start scenario from Pick:

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/common/scenario.py.

Use:

COMPONENT_PRESET = "pick_to_shelf_pick"

python examples/06_collect_component_task.py \
  --episodes 1 \
  --num-envs 1 \
  --save-end-scenario data/pick_to_shelf/nav_start.yaml

Load it into Nav and save a place-start scenario:

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/common/scenario.py.

Use:

COMPONENT_PRESET = "pick_to_shelf_nav"

python examples/06_collect_component_task.py \
  --episodes 1 \
  --num-envs 1 \
  --init-scenario data/pick_to_shelf/nav_start.yaml \
  --save-end-scenario data/pick_to_shelf/place_start.yaml

5. Define The Coherent Task

The coherent EnvCfg only selects the component tasks.

Source: src/ioailab/tasks/pick_to_shelf/config/g1/env_cfg.py.

GalbotG1PickToShelfEnvCfg = combined_task(
    name="GalbotG1PickToShelfEnvCfg",
    task_id="GalbotG1-PickToShelf-v0",
    phases=task_sequence(
        phase("pick", "GalbotG1-PickToShelf-Pick-v0"),
        phase("nav", "GalbotG1-PickToShelf-Nav-v0"),
        phase("place", "GalbotG1-PickToShelf-Place-v0"),
    ),
)

6. Run The Coherent Task

Use examples/07_compound_task.py.

Source files: examples/07_compound_task.py, src/ioailab/agents/flow/task_flow.py.

Keep the default preset active:

COMPOUND_AGENT_PRESET = "task_default"

Run:

python examples/07_compound_task.py \
  --task GalbotG1-PickToShelf-v0 \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1500

To override the phase agents in the example, switch one preset line:

# COMPOUND_AGENT_PRESET = "task_default"
COMPOUND_AGENT_PRESET = "pick_to_shelf_experts"

To evaluate trained PickToShelf pick/place policies, switch to:

# COMPOUND_AGENT_PRESET = "task_default"
# COMPOUND_AGENT_PRESET = "pick_to_shelf_experts"
COMPOUND_AGENT_PRESET = "pick_to_shelf_policy"

Use this only after replacing the example checkpoint paths in examples/07_compound_task.py with real trained pick/place policy checkpoints. The paths passed to PolicyAgent.from_checkpoint(...) are examples and can be customized:

phase_agents = {
    "pick": G1ManipulationPolicyActionAdapter(
        PolicyAgent.from_checkpoint("outputs/pick/model_best_training.pth")
    ),
    "place": G1ManipulationPolicyActionAdapter(
        PolicyAgent.from_checkpoint("outputs/place/model_best_training.pth")
    ),
}

Chapter 3: Build SortToShelf With Options and Sequence Agents

SortToShelf uses the same component-to-compound shape as PickToShelf, plus:

• --sorting-object
• a sequence agent for the standalone nav task

Objects:

red_cube
blue_cuboid
yellow_cylinder
green_cylinder

1. Run Pick Or Place

In examples/06_collect_component_task.py, activate:

# COMPONENT_PRESET = "pick_to_shelf_pick"
COMPONENT_PRESET = "sort_to_shelf_pick"

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/sort_to_shelf_pick/__init__.py, src/ioailab/tasks/sort_to_shelf_place/__init__.py.

Run Pick:

python examples/06_collect_component_task.py \
  --sorting-object red_cube \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000 \
  --dataset-path data/sort_to_shelf_pick_red_cube.hdf5

Run Place:

Switch the preset first:

COMPONENT_PRESET = "sort_to_shelf_place"

python examples/06_collect_component_task.py \
  --sorting-object red_cube \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000 \
  --dataset-path data/sort_to_shelf_place_red_cube.hdf5

2. Run Nav Sequence

SortToShelf Nav is one task, but the agent runs two steps:

drive base -> set place-start posture

In examples/06_collect_component_task.py, activate:

# COMPONENT_PRESET = "sort_to_shelf_place"
COMPONENT_PRESET = "sort_to_shelf_nav"

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/sort_to_shelf_nav/agent.py.

Run:

python examples/06_collect_component_task.py \
  --sorting-object red_cube \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1000 \
  --dataset-path data/sort_to_shelf_nav_red_cube.hdf5

3. Task Options

--sorting-object becomes task_options={"sorting_object": ...} in make_env(...).

Source files: examples/06_collect_component_task.py, src/ioailab/tasks/sort_to_shelf_pick/config/g1/env_cfg.py.

The option configures reset scenario, success terms, nav goal, and place posture. Keep object-specific task logic in the task option hook; the example only selects the component preset and passes --sorting-object.

4. Define The Coherent Task

The coherent SortToShelf task imports the three component tasks and the nav sequence agent.

Source: src/ioailab/tasks/sort_to_shelf/config/g1/env_cfg.py.

GalbotG1SortToShelfEnvCfg = combined_task(
    name="GalbotG1SortToShelfEnvCfg",
    task_id="GalbotG1-SortToShelf-v0",
    phases=task_sequence(
        phase("pick", "GalbotG1-SortToShelf-Pick-v0", fixed_base=True),
        phase(
            "nav",
            "GalbotG1-SortToShelf-Nav-v0",
            action_terms=("base", "legs", "left_arm"),
            agent=lambda env: nav_sequence_agent(
                sorting_object=(getattr(env, "task_options", {}) or {}).get(
                    "sorting_object", "red_cube"
                ),
            ),
        ),
        phase("place", "GalbotG1-SortToShelf-Place-v0"),
    ),
    actions_override=SortToShelfFullActionsCfg,
)

5. Run The Coherent Task

Use examples/07_compound_task.py.

Source files: examples/07_compound_task.py, src/ioailab/tasks/sort_to_shelf/config/g1/env_cfg.py.

Keep the default preset active:

COMPOUND_AGENT_PRESET = "task_default"

For SortToShelf, keep COMPOUND_AGENT_PRESET = "task_default". The existing pick_to_shelf_experts and pick_to_shelf_policy presets are PickToShelf-only; do not use them with GalbotG1-SortToShelf-v0 unless you add SortToShelf-specific phase-agent presets.

Run:

python examples/07_compound_task.py \
  --task GalbotG1-SortToShelf-v0 \
  --sorting-object red_cube \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1500

Collect full-task data:

python examples/07_compound_task.py \
  --task GalbotG1-SortToShelf-v0 \
  --sorting-object red_cube \
  --mode collect \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 1500 \
  --dataset-path data/sort_to_shelf_red_cube_full.hdf5

After red_cube works, repeat the same commands with the other object names.

Examples

Run GUI examples from make shell-gui; run headless data/training jobs from make shell. Each numbered example is a normal Python entry point.

Basic Pipeline

python examples/01_collect.py
python examples/02_mimic.py --task GalbotG1-PickCube-v0
python examples/03_train.py --task GalbotG1-PickCube-v0
python examples/04_eval.py --task GalbotG1-PickCube-v0 \
  --checkpoint outputs/pick_cube/model_best_training.pth --headless

01_collect.py shows the motion-planner path by default. It also contains commented blocks for TeleopAgent and final-scenario export. For GP001 teleop, use GalbotG1-PickCube-Teleop-v0 with TeleopAgent.from_device("gp001", task=task_id); rejected demos can be removed with dataset.drop() after an env.collect(...) candidate during done plus keep/drop/exit review.

Motion-planning collection is expert data generation. Expert tasks may have empty reward and curriculum managers because the planner, action stepping, and task termination define the episode boundary. Do not add dummy reward or curriculum terms only to produce manager summaries.

Component Tasks

Use examples/06_collect_component_task.py for standalone PickToShelf and SortToShelf phases. Select one COMPONENT_PRESET at the top of the file, then run the script.

PickToShelf presets target GalbotG1-PickToShelf-Pick-v0, GalbotG1-PickToShelf-Nav-v0, and GalbotG1-PickToShelf-Place-v0.

python examples/06_collect_component_task.py \
  --save-end-scenario data/pick_to_shelf/scenarios/nav_start.yaml

python examples/06_collect_component_task.py \
  --init-scenario data/pick_to_shelf/scenarios/nav_start.yaml \
  --save-end-scenario data/pick_to_shelf/scenarios/place_start.yaml

python examples/06_collect_component_task.py \
  --sorting-object red_cube \
  --save-end-scenario data/sort_to_shelf/scenarios/place_start_red_cube.yaml

GalbotG1-SortToShelf-Nav-v0 uses the task-local nav_sequence_agent: drive the base first, then set the place-start posture.

Coherent Tasks

Use examples/07_compound_task.py for full task flows. The default path uses task-owned phase agents; the file also shows how to override phase agents with planner or policy agents.

python examples/07_compound_task.py --task GalbotG1-PickToShelf-v0 --headless

python examples/07_compound_task.py --task GalbotG1-SortToShelf-v0 \
  --sorting-object red_cube --headless

python examples/07_compound_task.py --task GalbotG1-PickToShelf-v0 \
  --mode collect --dataset-path data/pick_to_shelf/full_expert.hdf5 --headless

Any BaseAgent can drive the same agent.act(env) -> env.step(action) loop: CuroboPlannerAgent, TeleopAgent, PolicyAgent, and TaskFlowAgent all return full IsaacLab action tensors.

ioailab Architecture

ioailab provides Galbot-specific task registrations, robot cfgs, action/sensor helpers, agents, dataset refs, and small tensor utilities. IsaacLab owns app launch, Gym env construction, managers, sensors, PhysX, recorder managers, and env.step(...).

The runtime boundary is explicit: ioailab configures IsaacLab; it does not hide IsaacLab.

Core Surface

task           = registered IsaacLab EnvCfg + scene + reset + MDP + task ID
component task = standalone task for one train/debug/eval phase
coherent task  = one normal task that runs a full long-horizon episode
phase          = row-local MDP state inside a coherent task
agent          = BaseAgent that returns full IsaacLab action tensors
scenario       = human-editable reset configuration for standalone starts
dataset        = DatasetRef over recorded HDF5/LeRobot artifacts

make_env(...) launches the app, registers tasks, and calls gym.make. The returned ioailabEnv stays transparent: callers can still use env.step(...), env.scene, env.unwrapped, sensors, and managers directly.

from ioailab.envs import make_env

env = make_env("GalbotG1-PickCube-v0", num_envs=1)

Top-level imports stay side-effect free. import ioailab must not register Gym IDs, launch Isaac Sim, touch assets, or import planner backends.

Long-Horizon Tasks

Long-horizon tasks use one coherent full task plus optional component tasks. PickToShelf is the reference shape:

GalbotG1-PickToShelf-v0
GalbotG1-PickToShelf-Pick-v0
GalbotG1-PickToShelf-Nav-v0
GalbotG1-PickToShelf-Place-v0

The coherent full task is a normal IsaacLab env:

reset once -> pick phase -> nav phase -> place phase -> full-task success

It must not create phase envs internally, restore external scene state between phases, or reset at intermediate phase boundaries. Physical continuity happens inside one live IsaacLab environment.

Component tasks are independent tasks with their own EnvCfg, MDP, success boundary, reset/default start, and default agent. They are used for collection, debugging, training, and evaluation.

Phase And Termination Rules

Phase state belongs to the coherent env and is row-local:

env 0: pick
env 1: nav
env 2: place
env 3: pick

Rows advance independently. A phase success predicate switches only that row to the next phase. The final phase success terminates the coherent episode. Any non-success termination from any phase remains a coherent task termination.

pick success  -> phase = nav
nav success   -> phase = place
place success -> episode success
timeout/fail  -> episode termination

Actions And Agents

The coherent task action space is the union of phase action spaces. Each phase controls only the terms it owns; inactive terms stay stable:

position term -> hold current joint position or previous target
velocity term -> zero velocity
gripper term  -> hold current gripper target/position
base term     -> zero velocity when inactive

TaskFlowAgent is a normal BaseAgent. It reads row phase from the env, calls the configured phase agent for each row group, and merges compact row actions into one full action tensor. It does not construct envs, call env.step(...), trigger resets, or load scene-state files.

SequenceAgent is the smaller primitive for ordered control inside one task phase. SortToShelf Nav uses it to drive the base first and then set the place-start posture.

Task Composition

Coherent tasks are declared by listing component tasks:

from ioailab.tasks.common.composition import combined_task, phase, task_sequence

GalbotG1PickToShelfEnvCfg = combined_task(
    name="GalbotG1PickToShelfEnvCfg",
    task_id="GalbotG1-PickToShelf-v0",
    phases=task_sequence(
        phase("pick", "GalbotG1-PickToShelf-Pick-v0"),
        phase("nav", "GalbotG1-PickToShelf-Nav-v0"),
        phase("place", "GalbotG1-PickToShelf-Place-v0"),
    ),
)

The component task packages remain the source of truth for component EnvCfg, MDP, success, and default agent facts. The coherent task imports or references them; component tasks must not depend on the coherent task package.

Scenarios

Scenarios are YAML reset-state overlays for assets that already exist in the task scene cfg. They are for standalone component starts and dataset/debug workflows.

Pick-v0  reset -> object on table
Nav-v0   reset -> object already held, robot at carry pose
Place-v0 reset -> robot near shelf, object held

Scenarios are not the normal connection mechanism for coherent tasks. Full tasks run continuously in one env.

Source Layout

src/ioailab/
├── robots/      # robot facts plus articulation/action/sensor cfgs
├── tasks/       # task IDs, scenes, EnvCfgs, MDP terms, component tasks
├── agents/      # BaseAgent, TaskFlowAgent, SequenceAgent, planners, policy, teleop
├── envs/        # make_env and ioailabEnv
├── datasets/    # DatasetRef, Mimic, LeRobot export
├── randomizers/ # reset-time domain randomizers
└── utils/       # asset lookup, logging, pose/tensor helpers

Task packages are task-first. Shared world geometry can live in task-local scene.py; robot-specific bindings live under config/<robot>/. There is no top-level ioailab.scenes package and no make_*_cfg scene factory layer.

Invariants

• Imports are side-effect free.
• ioailabEnv remains transparent over IsaacLab.
• Agents return action tensors and never own env stepping.
• Long-horizon physical continuity stays inside one coherent env.
• Intermediate phase success switches phase, not episode lifecycle.
• Component tasks stay independent and reusable.
• TaskFlowAgent and SequenceAgent are generic.
• Vectorized env rows advance asynchronously.
• Scenarios configure standalone starts, not coherent phase transitions.
• Motion planning uses cuRobo v2 (curobov2).

Action Agents & Task Flows

Action agents are dynamic action sources: they produce full IsaacLab action tensors for env.collect(...), env.evaluate(...), or a caller-owned loop.

dataset = env.collect(
    agent=agent,
    episodes=1,
    path="data/demo.hdf5",
)

Use env terminated/truncated signals, max_steps, or an explicit operator exit to define data boundaries. env.is_running() only guards the Isaac app lifecycle. For teleop, call env.collect(..., episodes=1) inside a human-review loop; dataset.drop() discards a rejected candidate.

Agent taxonomy

Reusable agents live under ioailab.agents; all share one IO shape (env in, optional env_ids mask in, full action tensor out):

Agents never implement cuRobo internals, env construction, or env.step(...) loops. A task-local motion plan returns ordered MotionSteps whose targets use one shared vocabulary — WorldTarget (absolute or computed) and AssetRelativeTarget (asset + offset, resolved against live scene state). Write the plan declaratively in YAML when it is a fixed waypoint sequence, or in Python (tasks/<task>/motion_plan.py) when it must compute from poses; both deserialize into the same MotionSteps. A plan bundles its planning config and is exposed through one task entry point. RL/IL cfg artifacts live under tasks/<task>/config/g1/agent_cfg/.

Task flows

TaskFlowAgent is the coherent-task agent used by PickToShelf. It reads the current row phase from the live env, groups rows by phase, calls each phase agent with env_ids, and merges those row actions into the full union action space while holding inactive action terms stable.

from ioailab.agents import TaskFlowAgent
from ioailab.envs import make_env

env = make_env("GalbotG1-PickToShelf-v0", num_envs=4)
agent = TaskFlowAgent.from_env(env)
dataset = env.collect(
    agent=agent,
    path="data/pick_to_shelf/full_expert.hdf5",
    episodes=36,
)

Advanced users can override any phase agent without changing the task:

agent = TaskFlowAgent.from_env(
    env,
    agents={
        "pick": pick_policy,
        "nav": custom_nav_agent,
        "place": place_policy,
    },
)

The task owns phase truth and success predicates; TaskFlowAgent does not construct envs, call env.step(...), trigger resets, or load external scene state for normal phase transitions.

Agent sequences

SequenceAgent is the generic primitive for ordered control inside one task phase. It runs a fixed list of agent_step(...) entries, advances each env row when the current step’s done predicate succeeds, resets the next step agent for only those rows, and composes compact step actions into the env’s union action space. TaskFlowAgent uses the same primitive under the hood for coherent full-task phases.

from ioailab.tasks.sort_to_shelf_nav.agent import nav_sequence_agent

agent = nav_sequence_agent(sorting_object="red_cube")

Use SequenceAgent when the env stays the same and only the active control strategy changes. Use TaskFlowAgent when a task cfg declares named phases and phase success boundaries.

Tasks

ioailab.tasks is an explicit IsaacLab-style registry of Galbot task IDs. It does not hide IsaacLab env construction, managers, sensors, or env.step(...).

Registered IDs

Create any registered task with make_env(...):

from ioailab.envs import make_env

env = make_env("GalbotG1-PickCube-v0", num_envs=1)

Component And Coherent Tasks

PickToShelf and SortToShelf use the same structure:

component tasks -> independent Pick/Nav/Place task IDs
coherent task   -> one continuous full-task env
agent           -> TaskFlowAgent dispatches phase agents by row phase

The coherent task does not rebuild envs, load external scene-state files, or reset between phases. It runs the full episode continuously. Component tasks are for standalone collection, debugging, training, and evaluation.

Override phase agents without changing the task:

from ioailab.agents import TaskFlowAgent

env = make_env("GalbotG1-PickToShelf-v0", num_envs=4)
agent = TaskFlowAgent.from_env(env, agents={"nav": custom_nav_agent})

Scenarios And Options

Nav and Place component starts use task-owned scenario YAML files under config/g1/scenarios/. Capture a final state with examples/06_collect_component_task.py --save-end-scenario ..., then load it with --init-scenario ... when intentionally replaying a standalone start.

SortToShelf selects the object through task_options={"sorting_object": ...} or the example flag --sorting-object. Valid values are:

red_cube
blue_cuboid
yellow_cylinder
green_cylinder

Package Layout

Each task package owns its task IDs, scene cfg, config/<robot>/env_cfg.py, MDP terms, registration metadata, optional task agents, and optional motion plans. Shared world geometry can live in task-local scene.py; robot-specific bindings, sensors, reset posture, and actions live under config/<robot>/. Robot-specific agent recipes live under ioailab.tasks.<task>.config.g1.agent_cfg.

There are no top-level scene modules or make_*_cfg scene factories. To author a task, copy an existing package such as ioailab.tasks.pick_cube and follow the Tutorial.

Data & Datasets

The imitation-learning data path runs through the public env, dataset, and policy APIs — collect, expand, train, evaluate:

from ioailab.agents import CuroboPlannerAgent
from ioailab.datasets import DatasetRef, mimic
from ioailab.envs import make_env
from ioailab.agents.policy import OptimizerCfg, Policy, RobomimicDiffusionTrainCfg

task_id = "GalbotG1-PickCube-v0"
env = make_env(task_id, num_envs=9, headless=True)

# 1. Collect — batch helper exports on env termination/truncation or max_steps.
agent = CuroboPlannerAgent.from_task(task_id)
dataset = env.collect(
    agent=agent, path="data/pick_cube_demos.hdf5", episodes=36, max_steps=1000
)


# Teleop uses an explicit review loop around env.collect(..., episodes=1);
# call dataset.drop() if the just-recorded candidate should be discarded.

# 2. Expand — IsaacLab Mimic, using the task stored on the dataset ref.
dataset = mimic(dataset, episodes=36)

# 3. Train — robomimic Diffusion Policy.
train_cfg = RobomimicDiffusionTrainCfg(
    output_dir="outputs/pick_cube",
    epochs=20,
    optimizer=OptimizerCfg(learning_rate=1.0e-4),
)
checkpoint = Policy.from_backend("robomimic_diffusion").train(
    dataset, train_cfg
)

# 4. Evaluate — load the checkpoint through the same policy backend.
agent = Policy.from_backend("robomimic_diffusion").load_checkpoint(checkpoint)
metrics = env.evaluate(agent=agent, episodes=36)

DatasetRef(path, task_id=...) carries the source task ID as provenance. Pass the registered task ID (e.g. GalbotG1-PickCube-v0) — the dataset helper resolves any Mimic-specific env (GalbotG1-PickCube-Mimic-v0) internally. Tasks own their EnvCfg, MDP terms, recorder config, and any Mimic metadata; there are no script bridges or handwritten dataset fallbacks.

LeRobot v3 export

The dev image installs the optional LeRobot v3 writer (lerobot==0.5.1, --no-deps so it cannot downgrade Isaac Sim’s curated numpy/torch/CUDA stack). Verify it inside the container:

python -c "from lerobot.datasets.lerobot_dataset import LeRobotDataset; print(LeRobotDataset)"

Export a staged HDF5 dataset with the explicit LeRobot submodule exporter:

from pathlib import Path

from ioailab.datasets.motion_plan_lerobot import MotionPlanLeRobotExporter

exported = MotionPlanLeRobotExporter(
    hdf5_path=Path("logs/lerobot/stack_cube_motion_plan_staging.hdf5"),
    lerobot_root=Path("logs/lerobot/stack_cube"),
).export()

The staging file sits beside the dataset root (not inside it), and the root must not already exist — LeRobot creates the directory structure itself. Exported features cover action, observation.state, and optional RGB image streams (observation.images.*) as LeRobot video features; depth/RGBD are not exported yet.

YOLO Segmentation Datasets

ioailab can also generate YOLO-seg datasets directly from task scenes using RGB-color masks by default, with Isaac semantic segmentation available as an explicit option. See docs/yolo_seg.md for the full pipeline (dataset generation, label visualization, model training, and inference).

Sensors

Cameras are IsaacLab config inputs. A task scene cfg activates the robot-mounted camera it needs; IsaacLab owns sensor creation, rendering, buffers, and runtime tensor reads. GalbotG1-PickCube-v0, GalbotG1-PickCube-Teleop-v0, and GalbotG1-PickToShelf-v0 request camera rendering through task metadata.

Activate a camera

Use the G1 sensor facade inside a task env_cfg.py:

from ioailab.robots.g1 import g1


@configclass
class MySceneCfg(InteractiveSceneCfg):
    robot = ...
    left_wrist_rgb_camera = g1.sensors.camera("left_wrist")
    front_head_rgb_camera = g1.sensors.camera("front_head")

Valid G1 mounts: front_head, left_wrist, right_wrist. The cfgs reuse calibrated mount transforms and intrinsics. There are no runtime camera-size or data-type flags; for custom non-G1 cameras, use IsaacLab camera cfgs directly.

Read at runtime

from ioailab.envs import make_env

env = make_env("GalbotG1-PickCube-Teleop-v0", num_envs=1)
env.reset()
left_rgb = env.scene["left_wrist_rgb_camera"].data.output["rgb"]
head_rgb = env.scene["front_head_rgb_camera"].data.output["rgb"]

GalbotG1-PickCube-Teleop-v0 owns the left-wrist and front-head RGB cameras and records them through its MDP observation cfg during recorder-backed collection. Use ioailab.utils.rerun_utils for optional Rerun viewer/URL helpers around recorded streams, outside task sensor cfgs. When depth is enabled on a custom camera, IsaacLab exposes it via keys such as distance_to_image_plane.

Robot Reference

Joint motion helpers

The action path stays explicit:

command -> ioailab tensor helper -> IsaacLab action tensor -> env.step(...)

Attach static action cfgs before env construction:

from ioailab.robots.g1.actions import g1_action_cfg

env_cfg.actions.leg_action = g1_action_cfg("legs", "absolute")

Pack named joints into the cfg-defined order at runtime:

from ioailab.robots.g1.actions import pack_g1_legs_absolute_joint_command

action_tensor = pack_g1_legs_absolute_joint_command(
    joint_names="leg_joint2", values=0.1, baseline=leg_rest_targets, env=env,
)
env.step(action_tensor)

G1 joint orders:

legs:  leg_joint1      -> leg_joint5       # (num_envs, 5)
left:  left_arm_joint1 -> left_arm_joint7  # (num_envs, 7)
right: right_arm_joint1-> right_arm_joint7 # (num_envs, 7)

Relative helpers fill unspecified joints with zero delta; absolute helpers need a baseline (or env) so unmentioned joints stay fixed. Packers accept env_indices — None applies to every row, an int/list/tensor to selected rows.

Grippers are bool open/close helpers returning shape (num_envs, 1):

from ioailab.robots.g1.actions import pack_g1_left_gripper_binary_command

gripper_action = pack_g1_left_gripper_binary_command(is_open=False, env=env)

Reusable runtime action-source facades live under ioailab.agents; task-local recipes live in tasks/<task>/config/g1/agent_cfg/motion_plan.py.

Robot assets

The canonical G1 USD is a local, gitignored file:

assets/galbot_one_golf_description/usd/galbot_one_golf.usda

Check it out from https://git.galbot.com/astra-synth/galbot_one_golf_description. Keep the bundle under assets/ and reference repository-local paths so the same path works locally and inside the container at /workspace/ioailab. Do not commit robot assets and do not add external asset-preparation or download workflows. Robot configs should not override USD-authored PhysX drive gains or limits by default. The planner-only mobile-base URDF is generated under assets/generated/ when cuRobo mobile-base planning needs it.

Developer Workflow

Development is Docker-first. Use the dev shell normally and the GUI profile when a visual Isaac Sim session (or GP001 teleop) is needed.

make build
make shell        # dev shell
make shell-gui    # GUI shell; auto-mounts serial teleop devices when attached

The compose file lives at docker/compose.yaml; prefer the Makefile targets over invoking Docker Compose directly. Inside the container, run code with python — it resolves to Isaac Sim’s runtime.

The Makefile derives the version-tagged Docker image from src/ioailab/__init__.py, so make build, make shell, make shell-gui, and validation targets use ioailab:<package-version> by default. Override ioailab_IMAGE, ioailab_IMAGE_REPOSITORY, or ioailab_IMAGE_TAG only when intentionally testing a non-release image.

For GP001 teleop, plug in the device and use make shell-gui. It mounts detected USB serial devices (/dev/ttyACM*, /dev/ttyUSB*); set GALBOT_GP001_DEVICE=/dev/serial/by-id/... to pick a specific one, or GP001_REQUIRED=1 to fail early when none is found. During collection, type done to finish the current candidate, then choose keep, drop, or exit at the review prompt — avoid Ctrl-C.

Validation

make format    # ruff format
make lint      # strict ruff check + advisory ty baseline
make typecheck # advisory ty over src, examples, tests
make test      # pytest

ty diagnostics are warnings while the IsaacLab/torch dynamic-type baseline is tightened. After Docker/dependency changes, rebuild with make build.

Pre-commit

Install local hooks from the dev extra when you want quick feedback before pushing a branch:

python -m pre_commit install
python -m pre_commit run --all-files

The hooks intentionally stay lightweight: ruff fixes/formatting plus basic YAML/TOML/whitespace checks. ty remains part of make typecheck, not a pre-commit hook, until the current dynamic IsaacLab/torch baseline is tightened.

Documentation

Docs are mdBook pages under docs/. Build or preview a single version with make docs / make docs-watch (both run mdbook on the host).

To publish all versions with the top-left version switcher, run:

make docs-versions   # host needs mdbook + git

This builds each version into book/<version>/ (the current working tree as latest, plus each released tag), writes a versions.json manifest read by the switcher (docs/theme/head.hbs), and adds a book/index.html that redirects to the current latest documentation. Serve book/ under the /ioailab/ subpath and deploy it manually. To add a release to the dropdown, prepend its tag to RELEASED_TAGS in scripts/build_versioned_docs.sh (only mdBook-era tags build; 0.0.1 used MkDocs and is excluded).

Rules

• Keep helper imports side-effect free.
• Use make_env(...) for env creation; env.step(...), managers, and sensors stay directly accessible on the returned ioailabEnv.
• Keep robot facts under ioailab.robots.<robot> and task code task-first under ioailab.tasks.<task>. Declare the robot-agnostic world in tasks/<task>/scene.py (a DefaultSceneCfg subclass) and layer the robot and sensors onto it in config/g1/env_cfg.py (no make_*_cfg scene factories, no top-level scene package). For very small tasks, the scene cfg may live directly in the robot-specific EnvCfg.
• Keep task packages task-first: motion recipes in tasks/<task>/config/g1/agent_cfg/motion_plan.py, Mimic helpers under tasks/<task>/mimic/, RL/IL cfg under tasks/<task>/config/g1/agent_cfg/. When a long task has component task IDs, give each phase its own task package and combine them through a task-flow cfg; never add a global phase package.
• For motion-planning examples and docs, use cuRobo v2 (curobov2).
• Do not add wrapper layers that hide or duplicate IsaacLab concepts.

The architecture source of truth is AGENTS.md plus Architecture. For the data pipeline and LeRobot export see Data & Datasets.