File size: 3,830 Bytes
09624c0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | ---
license: mit
base_model:
- lxsy/bitvla-bf16
tags:
- core-ai
- coreai
- on-device
- iphone
- apple-silicon
- ternary
- bitnet
- 1.58-bit
- vla
- vision-language-action
- robotics
- openvla
language:
- en
pipeline_tag: robotics
library_name: coreai
---
# BitVLA (1.58-bit Vision-Language-Action) — Core AI
The **first Vision-Language-Action model running fully on-device on iPhone**, via Apple **Core AI**.
A Core AI conversion of [`lxsy/bitvla-bf16`](https://huggingface.co/lxsy/bitvla-bf16)
([BitVLA, arXiv:2506.07530](https://arxiv.org/abs/2506.07530), MIT).
BitVLA takes an **image + a natural-language instruction** and predicts a **7-DoF robot
end-effector action** (Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper) — OpenVLA-style discrete action
tokens. Every transformer weight, in **both** the BitNet b1.58-2B language model **and** the
BitSigLIP-SO400M vision tower, is **1.58-bit ternary** ({-1, 0, +1}) — ~32× smaller than a
full-precision VLA (OpenVLA-7.5B ≈ 15 GB), so the whole policy fits and runs on a phone GPU. The
language model's per-layer linears run a **custom 2-bit packed-ternary Metal kernel**.
Part of the **Core AI model zoo** — on-device AI for iPhone & Mac through Apple Core AI:
https://github.com/john-rocky/coreai-model-zoo
## On-device (iPhone 17 Pro — Core AI GPU, greedy)
One image + instruction → 7-DoF action:
| stage | warm |
|---|---:|
| vision encode (BitSigLIP-SO400M, 256 tokens) | **0.13 s** |
| LLM prefill (≈308 positions, M=1 ternary kernel) | **8.8 s** |
| action decode (7 tokens) | **0.26 s** |
Resident ≈ 2 GB, no jetsam. On-device output matches the official model: **6/7 action tokens
identical**, 7-DoF action effectively identical, vision embeddings at per-token cosine **0.999**.
## What's in this repo
- **`h18p/`** — device-ready, AOT-compiled for the iPhone 17 Pro (h18p) GPU:
`bitvla_vision/` (BitSigLIP tower), `bitvla_llm_act/` (BitNet LLM, 256-row action head + ternary
kernel), `bitvla_device_data/` (preset-instruction text embeds + the 256-row action-token embed
table + `norm_stats` + a sample image — so the device needs no tokenizer or embedding table).
- **`aimodel/`** — portable source `.aimodel`s (vision + LLM); re-AOT for another device with
`xcrun coreai-build compile <…>.aimodel --platform iOS --preferred-compute gpu --architecture <arch>`.
## Architecture
- **LLM** = BitNet b1.58 2B4T (30L, hidden 2560, FFN 6912, GQA 20/5 hd128, ReLU² FFN, SubLN, RoPE
θ500000), W1.58-A8 (per-tensor absmean ternary weight + per-token int8 activation).
- **Vision** = BitSigLIP-SO400M (26L, hidden 1152, FFN 4304, patch14/224 → 256 tokens), ternary
linears; fp16 activations on device.
- **Connector** = 2-layer MLP (1152→2560→2560), fp16.
- **Action** = OpenVLA discrete: 256 vision embeds spliced into the LLM; 7 action tokens from the
vocab tail → 256-bin → BOUNDS-Q99 un-normalization (`norm_stats`, 27-dataset OXE mix;
e.g. `unnorm_key = bridge_orig`).
## Conversion & how it works
Recipe, kernel notes, and the device gotchas (custom kernel must be AOT-compiled — it can't JIT on
device; the dynamic-shape LLM `.aimodelc` loads with `expectFrequentReshapes=false`; vision uses
fp16 activations because the in-graph A8 quant stalls the GPU) are in the zoo:
- card: https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/bitvla.md
- knowledge: https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/bitvla-1.58bit-vla.md
- conversion: https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion
## License
MIT, inheriting [`lxsy/bitvla-bf16`](https://huggingface.co/lxsy/bitvla-bf16) /
[BitVLA](https://github.com/ustcwhy/BitVLA). This is a converted redistribution of the Core AI
artifacts; see the base model for original terms.
|