Document LeLab training and the patches it required
Browse files
README.md
CHANGED
|
@@ -42,16 +42,51 @@ Companion checkpoint for Part 4 of the book *VLA 로보틱스 입문서*.
|
|
| 42 |
language weights come from the pretrained base rather than being randomly
|
| 43 |
initialised.
|
| 44 |
|
| 45 |
-
##
|
| 46 |
|
| 47 |
-
|
| 48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
```
|
| 51 |
observation.images.top -> observation.images.camera1
|
| 52 |
observation.images.wrist -> observation.images.camera2
|
| 53 |
```
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
## Task string
|
| 56 |
|
| 57 |
The same sentence is stored in the dataset and must be supplied at inference:
|
|
|
|
| 42 |
language weights come from the pretrained base rather than being randomly
|
| 43 |
initialised.
|
| 44 |
|
| 45 |
+
## Trained with LeLab — and what had to be patched
|
| 46 |
|
| 47 |
+
Training ran through [LeLab](https://github.com/huggingface/leLab)'s web UI, not
|
| 48 |
+
the LeRobot CLI. Out of the box that does **not** produce a fine-tune of
|
| 49 |
+
`smolvla_base`, so the vendored LeLab used here carries three changes.
|
| 50 |
+
|
| 51 |
+
**1. Pass a base checkpoint.** LeLab builds its training command with
|
| 52 |
+
`--policy.type` only. That is correct for ACT, which has no pretrained base, but
|
| 53 |
+
for SmolVLA it means LeRobot constructs the architecture with randomly
|
| 54 |
+
initialised vision and language weights and trains only the action expert.
|
| 55 |
+
Nothing looks wrong while it happens — the loss falls, checkpoints are written,
|
| 56 |
+
inference runs — and LeRobot only warns about it on the PEFT path. The mismatch
|
| 57 |
+
shows up solely as `policy.pretrained_path: null` in `train_config.json`.
|
| 58 |
+
The patch adds a per-policy default and emits `--policy.path` instead.
|
| 59 |
+
|
| 60 |
+
**2. Use `=` form for policy arguments.** LeRobot's parser reads
|
| 61 |
+
`--policy.path` only as `--policy.path=<value>` (`parse_arg` matches on the
|
| 62 |
+
`--<name>=` prefix). Once a policy path is present, `get_cli_overrides("policy")`
|
| 63 |
+
also strips every `--policy.*` argument, so a space-separated
|
| 64 |
+
`--policy.device cuda` leaves a bare `cuda` behind and argparse rejects it. All
|
| 65 |
+
`--policy.*` arguments are emitted as `--policy.x=y`.
|
| 66 |
+
|
| 67 |
+
**3. Resolve the base to a local path, and align camera keys.** Two more things
|
| 68 |
+
break before training starts:
|
| 69 |
+
|
| 70 |
+
- On Windows, LeRobot turns `policy_path` into a `Path`, so the hub id
|
| 71 |
+
`lerobot/smolvla_base` becomes `lerobot\smolvla_base` and the hub lookup
|
| 72 |
+
fails. LeLab now resolves the base with `snapshot_download` and passes the
|
| 73 |
+
local snapshot directory.
|
| 74 |
+
- `smolvla_base` declares `camera1`/`camera2`/`camera3` as its image inputs,
|
| 75 |
+
while this dataset records `top`/`wrist`, which fails feature validation.
|
| 76 |
+
LeLab now reads the expected keys from the base `config.json`, reads the
|
| 77 |
+
dataset keys from `meta/info.json`, and generates a `--rename_map`.
|
| 78 |
+
|
| 79 |
+
For this run that produced:
|
| 80 |
|
| 81 |
```
|
| 82 |
observation.images.top -> observation.images.camera1
|
| 83 |
observation.images.wrist -> observation.images.camera2
|
| 84 |
```
|
| 85 |
|
| 86 |
+
Selecting **SmolVLA** in the LeLab training screen is enough; the base
|
| 87 |
+
checkpoint and the rename map are filled in by the backend. To train without a
|
| 88 |
+
pretrained base, pass an empty `policy_path`.
|
| 89 |
+
|
| 90 |
## Task string
|
| 91 |
|
| 92 |
The same sentence is stored in the dataset and must be supplied at inference:
|