Temporal-VLA — HAMLET-AAM on GR00T N1.6 / RoboMME

AAM = Action-aware Adaptive Memory. A memory module on top of the known-good HAMLET window architecture (GR00T-N1.6 backbone). It keeps a log-spaced 8-frame window of the past ([-1024,-512,-256,-128,-64,-32,-16,0] env-steps) and a learned gate that assigns a soft weight to each of the 8 memory slots every policy call — "how much to trust each past observation right now" — before they enter the memory transformer that conditions the action head. A second head predicts how the current observation follows from the (weighted) memory plus the last action chunk (action-conditioned forward prediction).

This bucket is everything you need to observe, debug, and trace the model at checkpoint-60000.

TL;DR result (this checkpoint, our eval harness, 10 eps/task)

The headline finding: the adaptive gate is genuinely good at selective recall (Permanence, where it ties HAMLET and doubles vanilla) but bad at temporal integration (Counting, where it falls below even no-memory). See memory_selection/HOW_TO_READ.md and the showcase/ clips.

A severe inference bug was found and fixed first (memory history was reset every policy call, collapsing the 8-frame window to 8 copies of the current frame). All results here are AFTER the fix. See THE_FIX.md.

How to navigate / trace

• showcase/ — start here. 5 curated episodes, each folder pairs:

  • rollout.mp4 — what the robot actually did (front + wrist cams; the red-bordered frames at the start are the demo the model "watched")
  • memory_heatmap.png — what the memory gate selected, aligned to the same episode
  • gate_weights_full_episode.jsonl — raw per-call gate weights for that task (trace it yourself) Folders: 01_PERMANENCE_success, 02_PERMANENCE_fail, 03_COUNTING_success, 04_COUNTING_fail, 05_IMITATION_fail.

• memory_selection/ — all 13 gate heatmaps (VideoUnmask, BinFill, MoveCube episodes) + HOW_TO_READ.md explaining the visualization and the Permanence-vs-Counting contrast.

• all_rollouts/<Suite>/<Task>/ — every one of the 160 rollout videos from the full 16-task eval, named ..._episode_N-{success|failure}.mp4. Browse any task's successes/failures.

• results_csv/ — the raw simulation_results.csv for all 16 tasks (col 3 = success bool).

• raw_memdumps/ — the full per-call gate-weight logs (JSONL) for the 3 instrumented tasks. Each line: {reset, t_hist, steps_back[8], weight[8]}. reset:true marks an episode start.

• THE_FIX.md — the inference bug and the one-line-ish fix, for reproducibility.

Reproduce a heatmap yourself

python scripts/viz_aam_memory.py <task_video_dir> <task_memdump.jsonl> <out_dir>

(script included as viz_aam_memory.py). Set AAM_MEMDUMP=/path/file.jsonl when running the eval to regenerate the raw gate logs.