Kimi-K3-REAP-448experts

A 50% expert-pruned, quality-unvalidated research artifact derived from moonshotai/Kimi-K3.

This is not Kimi K3. Half of the routed experts have been deleted and there has been no recovery training, so none of the base model's published benchmark results carry over — assume they do not hold here. If you want a working K3, use the original weights.

Compared with our earlier Kimi-K3-REAP-224experts, this checkpoint prunes half as aggressively and was selected with a calibration corpus that includes Chinese. In our own hands-on use it is the first of our three tiers that behaves like a usable model — see Hands-on impressions below, and note carefully what that section does and does not claim.

⚠️ What we do and do not know

1. No benchmark has been run on this checkpoint. None.

We have not measured SWE-bench, MMLU, perplexity, or anything else on these weights. Everything below is either a property of the surgery (verifiable), a subjective impression from our own use, or an expectation (not verified). Do not read "50% is safer than 75%" as "50% is safe", and do not read our impressions as scores.

What is known from the literature: we are not aware of a published result where a MoE survives 50% expert pruning without a healing phase and keeps its scores. Expect degradation.

2. There is no knee in the saliency curve at this ratio

Pruning to 448 experts discards 37.5% of the measured saliency mass (at 672 experts it would be 16.6%). The marginal cost per pruned expert rises monotonically through this region — 50% is a hardware-driven choice (it is what fits one 8×H200 node), not a point the measurements recommend.

3. Non-English: no longer visibly broken, but still not measured

Our 224-expert checkpoint is severely broken in Chinese — asked for a short story in Chinese it locks into a single-token loop, while its English reasoning in the same forward pass stays coherent. We traced that to the calibration corpus being >99% English: REAP scores an expert by its mean router-weighted activation over the tokens it actually sees, so experts specialising in an unseen language score near zero and are pruned first.

This checkpoint was selected with Chinese data added to the corpus (26% of calibration tokens), which measurably changes which experts survive — see below. On this checkpoint the collapse does not reproduce: the same prompts that break the 224 build return fluent, coherent Chinese here. That is a subjective observation on a handful of prompts, not a measurement of Chinese ability, and it is confounded — see the note at the end of the hands-on section. Treat non-English as "not visibly broken", not as validated.

4. Multimodal path is untouched but untested

Only text MoE experts were pruned; the vision tower is intact and we have never run an image through it. Assume nothing.

Hands-on impressions (not a benchmark)

We served this checkpoint and drove it with our own internal coding agent — a normal tool-calling loop over shell, file read/write/edit, and grep. This is subjective experience from a small number of sessions, not an evaluation. No task suite, no scoring, no repetitions, no baseline comparison. We are reporting it because "we never tried it" and "we tried it and it held up" are different states, and the card previously said the former.

What held up:

• Multi-step agentic work. A single session ran ~40 tool calls across roughly 35 model turns, building a small program from scratch, and stayed on task throughout. No turn-to-turn incoherence, no loss of the plan, no repair spiral of the kind we saw on the 224 build.
• Structured output. Every tool call in that session came back as valid JSON, and every call the agent issued executed without a malformed-arguments failure. K3's typed tool-call format deserves much of this credit.
• Chinese. Fluent and coherent, including the exact prompts that make the 224 build degenerate. We also scanned every assistant message in the session for degenerate repetition: the longest run of a repeated substring anywhere was 3 (a code-fence backtick). On the 224 build the same check finds runs in the thousands.
• Growing context. Prompt length grew past 30k tokens within one session with no degradation we could perceive.

What we did not establish: anything quantitative. We did not measure code correctness, instruction following, knowledge, math, long-context retrieval, or Chinese quality beyond "it is not degenerate". Nor did we compare against unpruned K3 on the same tasks — so we cannot tell you what the pruning cost. Assume it cost something.

⚠️ The Chinese result is confounded — do not cite it as proof the calibration fix works. Our 224 build differs from this one in two ways at once: the pruning ratio (75% → 50%) and the calibration corpus (English-only → Chinese-inclusive). Chinese working here is consistent with the corpus change being the cause, and our per-expert measurements predict the ratio change alone should not have fixed it (the affected experts sit at the bottom of the ranking at every cut point) — but that is a prediction, not a controlled result. The clean experiment is a 448 build selected with the English-only ranking, which we have not run.

The calibration-language effect, measured

This is the part of this artifact we think is genuinely worth publishing. Adding a Chinese corpus does not merely nudge the ranking — it changes which experts are kept, far beyond what adding the same amount of data can explain.

We compared three expert rankings computed from the same model and pipeline:

Ranking	Corpus
A	0.70 M tokens, English
B	1.42 M tokens, English (A plus a second English shard)
C	1.92 M tokens, English + Chinese (B plus 0.50 M Chinese tokens)

At 448 kept experts per layer, over all 92 MoE layers (41,216 kept slots):

Change	Slots whose keep/drop flips
A → B: +0.72 M English	1,222	2.96% — the noise floor
B → C: +0.50 M Chinese	3,971	9.63% — 3.2× the floor, from less added data

The Chinese-driven flips are not random churn near the cut point: 70% of the promoted experts are routed at least 1.5× more often (per token) by the Chinese corpus than by the English one, and their English support is well below the corpus median — that is, they were being under-measured rather than genuinely unimportant. The same pattern holds at 25% and 75% pruning.

Note what does not change: retained saliency mass at 448 is 62.6% under ranking B and 62.5% under C. The corpus changes which experts are kept, not how much mass survives. A language-blind corpus does not produce a visibly worse-looking pruning decision — which is precisely why this failure mode is easy to ship by accident.

We are not aware of this effect being documented in the expert-pruning literature, where evaluation is generally monolingual English.

What was and wasn't touched

Each of the 92 MoE layers keeps the 448 highest-saliency routed experts out of 896. Because K3 ships in native MXFP4 (mxfp4-pack-quantized, group_size=32), removing an expert is pure tensor surgery on the packed weights — nothing is dequantized or requantized, and the per-group scales of surviving experts stay bit-for-bit valid, since only whole experts are ever dropped.

Component	Treatment
Routed experts experts.{0..895}.w{1,2,3}	Pruned to 448, renumbered 0..447, packed weights copied verbatim
Router gate.weight, gate.e_score_correction_bias	Rows sliced to kept experts (bias is frozen at inference, so no requantile pass is needed)
Shared experts, latent routed_expert_up/down_proj	Untouched
KDA / MLA attention, AttnRes projections	Untouched
Norms, embeddings, lm_head, vision tower	Untouched

Kept experts stay in ascending original order, so router rows and expert indices remain consistent. K3 uses topk_group=1 (no grouped routing), so no group-balance bookkeeping is affected.

How the saliency was computed

REAP (router-weighted expert activation, mean-pooled over routed tokens). Scores are accumulated in the latent space (ℓ=3584): K3 applies routed_expert_down_proj before the per-expert loop, and the shared up-projection is identical across experts in a layer, so it cannot change their relative ranking.

Coverage	92 MoE layers (layer 0 is dense) × 896 experts, all finite, zero never-routed
Calibration tokens	1.92 M
Support per expert	median 30,012
Corpus	74% English (code-instruction, reasoning, agentic coding traces), 26% Chinese (code instructions, reasoning, technical and general Q&A)
Sequence length	32,768, left-truncated

Because a token reaches only 16 of 896 experts, per-expert support is thinner than we would like; matching the coverage of our 256-expert experiments would need ~3.7 M tokens. Read the ranking as well-resolved at the top and noisier near the cut.

Retained saliency mass by tier, from the same scores: 25% pruned → 83.4%, 50% (this model) → 62.5%, 62.5% → 50.4%, 75% → 37.0%.

Serving

837 GB of weights — this needs 8×H200 (1128 GB) or equivalent; it does not fit a 4×H200 node. Requires vLLM 0.27+ and trust_remote_code. At the time of writing the only working x86_64 path is the official image — the nightly wheels are aarch64-only.

docker run --gpus all -p 8000:8000 vllm/vllm-openai:kimi-k3-x86_64 \
  --model runrunway/Kimi-K3-REAP-448experts \
  --served-model-name k3-reap-448 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --trust-remote-code \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Notes from actually serving this build on 8×H200:

• The host NVIDIA driver must be ≥ 580 (CUDA 13.x) — the image ships torch 2.13+cu130 and fails late, after the weights download, on older drivers.
• --enable-prefix-caching must be passed explicitly; it is off by default for Kimi K3. Worth it: in an agent loop, which resends a long shared prefix every turn, we saw prefix cache hit rates near 90%.
• Expert parallelism is preferable to pure tensor parallelism here — it keeps each MXFP4 expert whole on one rank instead of sharding quantized matrices across ranks. 448 divides evenly by 8, 6 and 4, so EP stays available at all three node shapes.
• On Hopper the MXFP4 MoE dispatches to the Marlin W4A16 path (Using 'MARLIN' Mxfp4 MoE backend / Using MarlinExperts in the log), so weights stay 4-bit in memory rather than being upconverted. The cost is that there are no FP4 tensor cores on Hopper and the matmuls run at bf16 rates: we measured roughly 40 tokens/s single-stream generation. Expect Blackwell to be substantially faster.
• Weight load was ~108 GiB per rank (≈861 GiB total) in about 57 s, and the server was answering requests ~3 minutes after launch. During Marlin weight repacking vLLM logs a No available shared memory broadcast block found in 60 seconds warning — in our runs this was benign.
• K3's 69 KDA layers carry constant-size recurrent state, so KV is far cheaper than a dense-KV transformer: at a 128k context window we got a ~707k-token KV cache, and a 30k-token prompt used under 5% of it. There is room to raise --max-model-len well past the 32k we started with.
• If startup dies in flashinfer_autotune or cudagraph capture with cp_world_size must be positive, pass --no-enable-flashinfer-autotune and --attention-config '{"backend": "FLASHMLA"}'. Both were needed on Hopper.
• If you point a client at this endpoint, make the client's declared context window match your --max-model-len and check what output cap it actually puts on the wire. A client whose window bookkeeping disagrees with the server can land in a range where the server rejects the request but the client's own compaction has not triggered yet — which does not self-heal, it just fails every retry. This cost us an afternoon and was entirely our configuration, not the weights.

Other pruning ratios

Saliency is decoupled from the ratio — the same scores cut any tier without recomputation, and the pruning itself is CPU-only. If you want more headroom, 672 experts (25% pruned) discards only 16.6% of the saliency mass.

Our other published tiers, both from the same saliency scores:

Tier	Size	Fits	Our standing
448 (this model)	837 GB	8×H200	The one we would use. Held up in hands-on agent use.
384	734 GB	6×H200	Untested by us. Exists only because 6-GPU nodes were easier to get than 8.
224	475 GB	4×H200	Past the breaking point — we do not recommend using it. Kept up as a data point.

If you have 8 GPUs, take this one.

License and attribution

Derived from Kimi K3, © 2026 Moonshot AI, under the Kimi K3 License (included as LICENSE), which permits derivative works and their distribution provided the copyright and permission notice are retained. The base model, tokenizer, chat template, modeling code and all of the capability that survives here are Moonshot AI's work; this repository contributes only the expert selection and the surgery that applies it, and the defects described above are ours, not theirs.

Pruning criterion: REAP, from Cerebras' work on router-weighted expert activation pruning.

Provided as is, without warranty of any kind — see Section 5 of the license.