h2loop-ai/gemma-4-e2b-a16w8-hexagon

Gemma-4-E2B-it — A16W8 and A16W4 for the Qualcomm Hexagon NPU

google/gemma-4-E2B-it quantized to A16W8 (int16 activations, int8 weights) and compiled to QNN context binaries that run 100% on the Hexagon NPU — no CPU or GPU fallback.

Most shipped mobile builds of this model use int8 activations (GGUF-style q4_0). Keeping activations at 16 bits costs some memory and holds onto accuracy: on a held-out 400-question MMLU slice this build is statistically indistinguishable from the unquantized model.

Read the verification table before using a binary. Not every target below has been run on physical hardware, and this README states exactly which have. As of 2026-07-27 both prefill binaries are verified on both v79 and v81 silicon; v81 decode is not verified.

---

Verification status per target

Everything below was measured on physical hardware — Snapdragon 8 Elite (SM8750, HTP v79) via Qualcomm Device Cloud, device 41c5710f, 2026-07-26, and Snapdragon 8 Elite Gen 5 (SM8850, HTP v81) via Qualcomm AI Hub inference, 2026-07-27. Each table says which.

v79 (hexagon-v79 / SM8750 / Snapdragon 8 Elite) — verified

binary	role	on hardware	perf
gemma4_decode_wgqa_int8kv_a16w8_v79.bin	decode (recommended)	✅ exact float match	65.3 ms/step · 15.3 tok/s
gemma4_decode_wgqa_a16w8_v79.bin	decode (no int8-KV)	✅ exact float match, deterministic	69.8 ms/step · 14.3 tok/s
gemma4_trunk_a16w8_v79.bin	prefill, fixed SEQ=128	✅ 12/12 next-token, cos 0.992	—

Decode — the generated text is token-for-token identical to the float ONNX reference:

prompt : "The capital of France is"   (Gemma-4 chat template)
device : 'The capital of France is **Paris**.'
float  : 'The capital of France is **Paris**.'

The/ capital/ of/ France/ is/ **/Paris/**., terminating correctly on <turn|>. Two consecutive runs produced byte-identical output, so decoding is deterministic on device.

Prefill — 12 held-out chat prompts, each one forward pass, compared against float: 12/12 (100%) next-token top-1, hidden cosine mean 0.992 / min 0.986.

Speed ladder (v79, A16W8, AI Hub profile on real hardware)

decode graph	ms/step	tok/s	speedup
naive full-KV	307.9	3.25	1.0×
+ windowed KV + broadcast-GQA	69.8	14.3	4.41×
+ int8-KV (full-attention slots)	65.3	15.3	4.71×

int8-KV cost no accuracy on the held-out check (32/32 content-token agreement, unchanged).

v81 (hexagon-v81 / SM8850 / 8 Elite Gen 5) — prefill verified, decode still untested

Measured on real Snapdragon 8 Elite Gen 5 silicon via AI Hub inference, 2026-07-27.

binary	role	on hardware	perf
gemma4_trunk_a16w8_v81.bin	prefill, fixed SEQ=128	✅ 11/12 next-token, cos mean 0.991 / min 0.981	—
gemma4_decode_wgqa_int8kv_a16w8_v81.bin	decode	⚠️ still unverified — see below	profiles at 57.4 ms/step (17.4 tok/s)

Prefill on v81 matches v79 bit-for-bit. The 12 held-out prompts were run through the v81 trunk binary, and the same harness was run against the v79 trunk binary as a control, so the two are directly comparable rather than being compared across measurement paths:

trunk binary	device	prompts	next-token top-1	hidden cos (mean / min)
gemma4_trunk_a16w8_v81.bin	SM8850 (v81), AI Hub	12	11/12	0.99077 / 0.98050
gemma4_trunk_a16w8_v79.bin	SM8750 (v79), AI Hub	2 (control)	1/2 — same prompt flips, same cosines	0.99234, 0.98050
gemma4_trunk_a16w8_v79.bin	SM8750 (v79), adb / Device Cloud	12	12/12	0.992 / 0.986

The v79 rows are a control on the measurement path, not a second verification: the AI Hub row deliberately re-ran only the harness-validation prompt and the one prompt v81 flipped.

The single v81 disagreement is "Name a planet with rings.", where float opens '**' and the device opens 'The' at cos 0.9805 — a near-tie between two plausible sentence openings, not a degradation. Re-running that exact prompt on the v79 binary reproduced the identical mismatch at an identical cosine of 0.9804982542991638, so it is a property of the A16W8 quantization, not of v81. On this evidence the two architectures are numerically indistinguishable on the trunk.

Decode on v81 remains unverified. An earlier spot-check of the v81 decode binary showed hidden cosine degrading 0.912 → 0.848 → 0.794 → 0.839 across prefill steps 0–3, with the hardware norm about half of float at step 2, and was stopped before the token comparison. That result has not been explained or reproduced, and the trunk result above does not clear it: the trunk is a different graph. Two candidate explanations remain open — the decode harness itself, and the fact that the v81 decode binary was compiled from the int8-KV export (decode_wgqa_A16W8_int8kv_full) whereas the token-exact v79 verification used the plain WGQA export. Treat gemma4_decode_wgqa_int8kv_a16w8_v81.bin as "failed a spot-check, cause unknown" — not as broken, and not as usable.

Why decode is expensive to verify: Qualcomm Device Cloud provisions v79 parts only, so there is no adb-attached v81 device, and AI Hub inference bills one farm job per decode step (~23 jobs for one short sentence). The trunk is stateless — one forward per prompt — which is why prefill could be verified for 12 jobs and decode was not.

If you have v81 hardware, the useful next step is decode: run the loop per How to run and compare against float. If it diverges, re-verify against a v81 build of the non-int8-KV export to isolate whether int8-KV interacts badly with v81, and check that your QAIRT install ships a hexagon-v81 skel matching the compile.

What is not verified

• Long-context / ring-wrap. The windowed graph uses 512-entry ring buffers on the sliding layers. Prompts long enough to wrap the ring (>512 tokens) were not exercised on hardware. Verified prompts were ~14–18 tokens.
• Prefill beyond 128 tokens. The trunk graph is a fixed SEQ=128 window. Longer prompts are not covered by it at all.
• Generative benchmarks (GSM8K etc). Not measured. MMLU is one forward pass per question; multi-step generative reasoning compounds error over hundreds of steps and is untested here.
• Throughput is NPU inference time from an AI Hub profile job, not end-to-end tokens/s. Host-side embedding lookup and lm_head are excluded; a real application adds those, and the net-run harness used for correctness reloads the context each step so its wall-clock (~3.6 s/step) is not a throughput number.

---

Accuracy

Held-out MMLU, 0-shot, chat-formatted, 400 questions disjoint from all calibration data:

	accuracy
base model (float, ≡ HF)	56.75% ± 2.48
A16W8 (this build)	59.25% ± 2.46
delta	+2.50 pp
random baseline	25.00%

The +2.50 pp delta is about one standard error — not evidence that quantization improves the model. The correct reading is that A16W8 costs no measurable MMLU accuracy. Note the two models disagree on ~24% of individual questions; they match in aggregate, not per-question.

---

Two things you must get right

1. Use the chat template. The raw completion format makes this instruction-tuned model degenerate. Verified on the unquantized model, so this is not a quantization artifact:

format	output
raw + greedy	' France is France is France is…'
raw + temperature / top-p	byte-identical degeneration
raw + repetition_penalty 1.2	byte-identical degeneration
chat template + plain greedy	'The capital of France is **Paris**.'

Token layout (verified byte-exact against transformers.apply_chat_template):

[2 <bos>, 105 <|turn>, 2364 'user', 107 '\n'] + PROMPT + [106 <turn|>, 107, 105, 4368 'model', 107]

Stop generation on 106 (<turn|>) or 1 (<eos>).

2. Match the mask constant. Attention masks use a finite NEG = -1e4, not -inf or float32.min. -inf cannot survive int16 activation quantization — it blows out the range so real scores round to zero. -1e4 still zeroes the softmax while leaving real scores resolved. The host must use the same value the model was calibrated with.

---

Architecture

The graph is split so the >2 GB vocab tensors never enter it:

• Host (CPU/ARM): token embedding lookup, per-layer embedding lookup, and the tied lm_head with 30·tanh(x/30) softcap.
• NPU: the transformer decode graph, with the KV cache resident on device.

Gemma-4-E2B is dense: 35 layers, hidden 1536, GQA 8 query heads → 1 KV head, head_dim 256, 262144-token vocab, and hybrid attention (28 sliding-window layers of window 512, interleaved with 7 full-attention; KV shared across the last 20 layers, so only 15 layers store KV).

Why the fast graph is fast

Decode is KV-attention-bound, not weight-bound. The published binary uses two changes over a naive full-KV decode graph:

• Windowed KV — sliding-window layers use a 512-entry ring buffer instead of a full 4096 buffer; only the 3 full-attention layers keep 4096. KV traffic per step drops from ~288 MB to ~63 MB. The ring index is computed inside the graph as cache_position % buf, so the host just passes pos.
• Broadcast GQA — the expand op that materialized 1 KV head into 8 copies is removed (verified: 0 Expand nodes in the exported ONNX).

Net effect on v79: 307.9 ms → 69.8 ms per decode step (4.41×).

Credit: these two levers come from the tps/ work in gemma-4-e2b-hexagon-npu — this repo contributes a corrected quantization of that graph.

---

Files

gemma4_decode_wgqa_int8kv_a16w8_v79.bin  1.9 GB  decode, v79  (recommended)
gemma4_decode_wgqa_a16w8_v79.bin         1.9 GB  decode, v79, no int8-KV
gemma4_trunk_a16w8_v79.bin               1.9 GB  prefill, v79, fixed SEQ=128
gemma4_decode_wgqa_int8kv_a16w8_v81.bin  1.9 GB  decode, v81   [UNVERIFIED - see above]
gemma4_trunk_a16w8_v81.bin               1.9 GB  prefill, v81  (verified on v81 silicon)

host-model/embed_tokens_weight.bf16            769 MB  token embeddings
host-model/embed_tokens_per_layer_weight.bf16  4.4 GB  per-layer embeddings
host-model/tokenizer.json                       31 MB
host-model/norm_weight.bf16                              final norm (diagnostics)

runtime/hostlib.py               host embeddings, chat template, lm_head + softcap
runtime/run_gate.py              host orchestrator (the autoregressive loop)
runtime/verify_trunk.py          prefill checker vs a float reference
runtime/stage_device.sh          push everything to an adb device
runtime/gate_ondevice_wgqa.sh    on-device decode step + KV rotation
runtime/gate_ondevice_int8kv.sh  same, int8-KV binary
runtime/gate_ondevice_trunk.sh   on-device prefill pass (no KV)
requirements.txt

The three host-model tensors are ~5.2 GB and stay on the host by design — putting the 262144-token vocab in the graph blows past ONNX's 2 GB protobuf limit.

---

How to run

What you need that is NOT in this repo

You cannot run this from this repo alone. One dependency is missing by necessity:

1. Qualcomm AI Engine Direct (QAIRT / QNN) SDK — supplies qnn-net-run and libQnnHtp*.so plus the HTP Stub/Skel pair for your Hexagon version. These are Qualcomm-licensed and not redistributable here, so you must install the SDK yourself (free, from Qualcomm). Built and tested against QAIRT 2.45.
   • v79 needs libQnnHtpV79Stub.so + libQnnHtpV79.so / libQnnHtpV79Skel.so
   • v81 needs the V81 equivalents. Check your SDK actually ships hexagon-v81; older installs do not.
2. A device: Snapdragon 8 Elite (SM8750, v79) or 8 Elite Gen 5 (SM8850, v81), reachable over adb. Qualcomm Device Cloud works — that is what this was verified on.
3. Host Python 3.9+ with numpy and tokenizers (pip install -r requirements.txt). No torch, no transformers needed to run — only to reproduce the quantization.
4. ~6 GB free on the host for the embedding tensors, ~4 GB free on the device (/data/local/tmp) per pair of context binaries.

Step 1 — get the repo

pip install -r requirements.txt
git lfs install
git clone https://huggingface.co/h2loop-ai/gemma-4-e2b-a16w8-hexagon
cd gemma-4-e2b-a16w8-hexagon

Step 2 — connect the device

adb devices -l          # confirm your serial

On Qualcomm Device Cloud, tunnel the adb server first, then point adb at it:

ssh -i <your-qdc-key>.pem -L 5037:<QDC_HOST>:5037 -N sshtunnel@ssh.qdc.qualcomm.com &
export ADB_SERVER_SOCKET=tcp:127.0.0.1:5037
adb devices -l

Never run adb kill-server against that tunnel — it kills the remote pod's adb server, which you cannot restart without portal access.

Step 3 — stage onto the device

export QAIRT_DIR=/path/to/qairt/2.45.0.xxxxxx     # your SDK install
./runtime/stage_device.sh <serial> v79             # or: v81

This pushes qnn-net-run, the HTP libs/skels, the matching *_v79.bin context binaries, and the on-device step scripts. Two 1.9 GB pushes over adb take a while; over a QDC tunnel a single stream runs ~1 MB/s, so expect ~30 min unless you parallelise (see Slow adb below).

KV buffers are not pushed — run_gate.py creates them on device with dd.

Step 4 — run the autoregressive loop

python runtime/run_gate.py \
  --prompt "The capital of France is" \
  --ntokens 14 \
  --adb-serial <serial> \
  --chat --wgqa \
  --script gate_ondevice_int8kv.sh

Expected output:

continuation: 'The capital of France is **Paris**.'

Flags that matter:

flag	why
--chat	required. Without it the -it model degenerates into ' France is France is …'
--wgqa	required for these binaries — selects 512-entry ring buffers and the 512-wide sliding mask
--script	pick the binary: gate_ondevice_int8kv.sh (recommended) or gate_ondevice_wgqa.sh

Drop --script to use the non-int8-KV binary.

Step 5 (optional) — check prefill

verify_trunk.py compares the trunk against a float reference. Producing that reference needs torch + transformers on a host that knows the gemma4 architecture (transformers ≥ 5.12 — older versions raise KeyError: 'gemma4'), so it is a reproduction step rather than part of normal use.

---

A16W4 (int16 activations, int4 weights)

A second precision in the same repo: AWQ per-channel int4 weights with the same int16 activations and the same WGQA decode graph. It is half the size and ~2× the NPU compute speed of A16W8 — and it costs real accuracy. Unlike the A16W8 build, this one is not statistically indistinguishable from the unquantized model.

	A16W4	A16W8
decode + prefill binaries	1.80 GiB	3.54 GiB
decode NPU compute	33.5 ms/step · 29.9 tok/s	65.3 ms/step · 15.3 tok/s
held-out MMLU (n=400)	48.50% ±2.50	59.25% ±2.46
vs unquantized base (56.75% ±2.48)	−8.25 pp — a real loss (~2.3 SE)	+2.50 pp (~1 SE, not significant)

Choose A16W4 only if you are memory-constrained. If you can afford 3.5 GiB, A16W8 is float-exact on device and this one is not.

A16W4 verification status

v79 (SM8750 / Snapdragon 8 Elite) — verified on hardware

Measured on a physical Snapdragon 8 Elite via Qualcomm Device Cloud, device 2a38935c.

binary	role	on hardware
gemma4_decode_awq_pc_int4_v79.bin	decode	✅ runs, deterministic; token-exact on some prompts, not all
gemma4_trunk_awq_pc_int4_v79.bin	prefill, fixed SEQ=128	✅ 10/12 next-token, cos mean 0.818 / min 0.740

Decode, free-running with the chat template, against the float reference:

"The capital of France is"          -> 'The capital of France is **Paris**.'          exact  8/8
"Who wrote Hamlet?"                 -> '**William Shakespeare** wrote *Hamlet*.'      exact  8/8
"What color is grass?"              -> 'Grass is **green**.'                          2/6
                                       (float: 'Grass is typically **green**.')
"Name a country in South America."  -> 'Name a country in South America.'             0/7
                                       (float: 'Brazil')

Two consecutive runs of the first prompt were byte-identical, so decoding is deterministic.

Read the last row honestly. On that prompt the model echoed the input instead of answering. That is a genuine failure, not a paraphrase, and it is the kind of degradation the −8.25 pp MMLU predicts. Two of four prompts are token-exact with float; one differs only in wording; one fails outright.

Prefill — 12 held-out chat prompts vs float: 10/12 (83.3%) next-token top-1, hidden cosine mean 0.818 / min 0.740. (A16W8 on the same harness: 12/12, cos 0.992 / 0.986.) The device result matches this build's CPU simulation (11/12, cos 0.813) within noise, i.e. the hardware reproduces the quantization faithfully — the loss is in the quantization, not the port.

v81 (SM8850 / 8 Elite Gen 5) — prefill verified, decode never run

Measured on real Snapdragon 8 Elite Gen 5 silicon via AI Hub inference, 2026-07-27.

binary	role	on hardware
gemma4_trunk_awq_pc_int4_v81.bin	prefill, fixed SEQ=128	✅ 9/12 next-token, cos mean 0.809 / min 0.610 — bit-identical to v79
gemma4_decode_awq_pc_int4_v81.bin	decode	⚠️ never run on hardware

The v81 and v79 trunks produced bit-identical output. Both binaries were run through the same harness on their own silicon — v81 on SM8850, v79 on SM8750 — over the same 12 held-out prompts. Every one of the 12 hidden cosines agreed to the full float32 bit pattern (maximum absolute difference across all 12: exactly 0.0), and all 12 device tokens matched, including the same three disagreements with float on the same three prompts:

trunk binary	device	next-token top-1	hidden cos (mean / min)
gemma4_trunk_awq_pc_int4_v81.bin	SM8850 (v81)	9/12	0.80949 / 0.61016
gemma4_trunk_awq_pc_int4_v79.bin	SM8750 (v79)	9/12	0.80949 / 0.61016

HTP consumes per-channel int4 natively on both architectures and, on this graph, does so deterministically and identically. The v81 trunk carries exactly the A16W4 accuracy loss documented above — no more, and no less.

These 9/12 figures come from the AI Hub inference path; the 10/12 (cos 0.818 / 0.740) quoted for v79 above was measured over adb on Device Cloud with the same prompts and harness logic. The one-prompt difference is the near-tie on "Who painted the Mona Lisa?" resolving differently between the two measurement paths, not an architecture effect — which is precisely why the v79 control was re-run here rather than compared across paths.

Decode on v81 has still never been run. Same reason as A16W8: AI Hub bills one farm job per decode step, and Device Cloud has no v81 part. Given that the trunk is bit-identical across the two architectures, decode is likely fine too — but that is an inference, not a measurement, and A16W4 decode is the path where this precision's real failures show up (it echoes the prompt instead of answering on at least one prompt, on verified v79 silicon).

Reproducing the A16W4 quantization

AIMET QuantizationSimModel, param_type=int4, activation_type=int16, quant_scheme=min_max, NEG=-1e4, calibrated on real activations captured from chat-formatted decode loops — every caveat in the A16W8 section applies here unchanged.

Weights additionally use AWQ per-input-channel scaling folded into the preceding RMSNorm, so it costs nothing at inference. Sanity check: the AWQ float trunk scores exactly the same MMLU as the unquantized base (56.75%), confirming the fold is numerically exact.

Two things that will waste your time

IO dtype depends on who compiled the binary. AI Hub builds expose QNN_DATATYPE_FLOAT_32 graph inputs; binaries produced locally by qnn-context-binary-generator keep the DLC's QNN_DATATYPE_UFIXED_POINT_16. The reference harness writes fp32, so it is correct only for AI Hub builds — feed fp32 into a uint16-IO binary and you get fluent-looking garbage. Check before debugging anything else:

qnn-context-binary-utility --context_binary X.bin --json_file io.json   # -> graphInputs[].dataType

Pair each binary with the QAIRT version that built it. A 2.48-compiled context binary on a 2.42 runtime fails with Could not create context from binary — the same message as an HTP arch mismatch, but a completely different cause.

A note on LPBQ / group-64 int4

We built and measured it; do not use it. Low-Power Blockwise Quantization stores 4-bit values inside an int8 container, so it saves no memory at all (1.78 GiB — larger than A16W8's 1.77 GiB), scores 6 pp worse than A16W8 on MMLU, and runs 9.3× slower (609 ms/step) because HTP cannot consume blockwise weights natively and inserts ~1079 dequantize ops per step. It is dominated by A16W8 on every axis. Per-channel int4 is the only 4-bit form worth shipping on this graph.

A16W4 limitations

Everything in the A16W8 limitations section applies, plus:

• Not float-equivalent. It fails some prompts outright (see the table above).
• The speed figure is NPU compute per step, not TPOT. Real end-to-end throughput is unmeasured and requires a resident-KV runtime. Our file-based qnn-net-run harness measures 598 ms/step for A16W4 and 592 ms/step for A16W8 — nearly identical despite A16W8 having twice the compute — because ~126 MB of KV file traffic per step dominates everything. Do not quote either number as throughput.
• v81 prefill is verified and bit-identical to v79; v81 decode has never been run.

---

How the loop actually works

run_gate.py is the reference implementation, and deliberately simple:

1. Tokenize with the chat template (hostlib.encode_chat).
2. Zero the KV buffers on device.
3. For each position: look up embeddings on the host, write the six small per-step tensors, run qnn-net-run once on device, pull back hidden (1536 floats).
4. Apply the tied lm_head + 30·tanh(x/30) softcap on the host, take the argmax.
5. The on-device script renames present_* → past_* so KV never crosses adb.
6. Stop on <turn|> (106) or <eos> (1).

This harness is for correctness, not speed. It re-loads the 1.9 GB context binary every step, so its wall clock (~3.6 s/token) is ~50× worse than the NPU's actual 65 ms. A real application loads the context once, keeps KV device-resident, and does embeddings + lm_head in-process. Building that is left to you.

Slow adb

A single adb stream over a QDC tunnel is bandwidth-delay-product limited (~1 MB/s), not bandwidth limited. Splitting the binary and pushing chunks over separate SSH tunnels (one local port each) reached ~7 MB/s:

split -n 6 -d gemma4_decode_wgqa_int8kv_a16w8_v79.bin chunk.
# ...one `ssh -L 503X:$HOST:5037` per chunk, then push each with its own
#    ADB_SERVER_SOCKET=tcp:127.0.0.1:503X, then on device:
adb shell 'cd /data/local/tmp/gemma/artifacts && cat chunk.* > out.bin && rm chunk.*'

Verify the checksum afterwards (SHA256SUMS) — and wait for all pushes to finish before concatenating, or you will silently assemble a truncated file.

Troubleshooting

symptom	cause
Could not create context from binary	HTP arch mismatch — a v81 binary will not load on a v79 device, or vice versa
Cannot assign data from unexpected type. Expected int32, got int64	binaries are built with --truncate_64bit_io, so position_ids/cache_position are int32
Output repeats ' France is France is …'	--chat missing
Fluent but wrong answer	binary/mask mismatch — the host NEG must be -1e4, matching calibration
Garbage tokens, hidden norm ≈ 0	wrong context binary, or KV buffers not zeroed before position 0
Different output across identical runs	a KV buffer was corrupted mid-push; re-seed (dd on device) and retry
Fluent garbage from an A16W4 binary you compiled yourself	local qnn-context-binary-generator emits UFIXED_POINT_16 IO; the harness writes fp32. Use the AI Hub build, or check graphInputs[].dataType
Could not create context from binary on a binary that used to work	QAIRT runtime version does not match the version that compiled it

Reproducing the quantization

AIMET QuantizationSimModel, param_type=int8, activation_type=int16, quant_scheme=min_max, calibrated on real activations captured from chat-formatted decode loops — not random noise, and not raw-format text.

Both of those details matter and each caused a distinct on-device failure:

• Calibrating on np.random.randn produced a binary whose residual stream collapsed to zero on hardware (final hidden norm 0.0000 → pure noise tokens), even though it compiled fine.
• Calibrating on raw-format text and then running chat-format prompts produced fluent but unfaithful output on hardware — the chat template's special tokens hit activation ranges the quantizer never observed.

min_max outperformed tf_enhanced here: on int16 there are 65k levels, so range coverage matters more than outlier clipping, and tf_enhanced mis-estimated the range badly enough to inflate hidden norms ~10×.

Limitations

• Fixed 4096 context.
• Prefill is not included in this repo; the decode graph can prefill token-by-token, which is slow for long prompts.
• Batch size 1 only.
• --truncate_64bit_io at compile time means index inputs (position_ids, cache_position) are int32 on the compiled binary, though the float ONNX takes int64.