Haiku-base

Pretrained base model for the Haiku family (~655M)

A larger TinyGDN hybrid: Kimi Delta Attention memory plus gated multi-head latent attention

---

What this is

Haiku-base is the pretrained (base) checkpoint for Haiku, the ~655M successor to the Tercet family.

• Scales kerzgrr/Tercet-base from ~502M to ~655M parameters
• Hybrid Kimi Delta Attention (KDA) recurrent layers + gated MLA (NoPE) full-attention layers
• Own 65,536 BPE tokenizer (not the Tercet 49k vocab)
• This repo is pretrain-only raw text continuation
• Chat / instruction SFT is not released

This base model is for continuation and research. It will not follow instructions reliably.

---

Model Architecture

Pipeline: Text Prompt → BPE-65K Tokenizer → Haiku Hybrid Decoder (36L) → Next-token Prediction

Hybrid block schedule (×36)

Every 4th layer is gated MLA; the rest are Kimi Delta Attention:

KDA, KDA, KDA, MLA, … (3:1 recurrent-to-attention)

Component	Details
Kimi Delta Attention	Linear-time recurrent memory (flash-linear-attention)
Gated MLA	DeepSeek-style latent KV, content-only QK (NoPE), full-rank output gate
MLP	SiTU-GLU
Residuals	Block attention residual
Norm	Zero-centered RMSNorm
Embeddings	Tied input / output

Technical specifications

Architecture	Haiku hybrid (KDA + gated MLA)
Parameters	655,270,488 deployable
Hidden size	1,024
Intermediate (MLP)	3,840
Layers	36 (27 KDA + 9 gated MLA)
Attention	8 heads, Q LoRA rank 512, KV LoRA rank 256
Linear (KDA)	8 heads × 128 dim
Context (trained)	2,048
Max position embeddings	32,768
Vocabulary	65,536 (BPE)
RoPE θ	1,000,000 (partial factor 0.5; used by KDA)
Precision (Hub weights)	bfloat16 EMA
Weight file	model.safetensors (~1.22 GiB)

---

Training (pretrain)

Dataset	FineWeb-Edu (10.13B packed train tokens)
Tokens seen	4,404,019,200
Sequence length	2,048
Objective	Next-token prediction (+ MTP during training; not used at decode)
Optimizer	Hybrid Muon + AdamW — β₁=0.9, β₂=0.95
Peak LR	2 × 10⁻⁴
Warmup	1% of steps
Grad clip	1.0
EMA	Karras power EMA (γ=1.0, p=0.75, max decay 0.9999) — this Hub file is the EMA weights
Checkpoint	optimizer step 8,400
Val loss (EMA)	3.6904 (ppl 40.06)

---

Install

1) System requirements

• Python 3.10+
• CUDA GPU strongly recommended
• PyTorch with CUDA matching your driver

2) Create an environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

pip install torch

4) Install Python deps

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by inference.py on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

5) Download the inference script

curl -L -o inference.py https://huggingface.co/kerzgrr/Haiku-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Haiku-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo.

---

Quick start

Single prompt (streams tokens):

python inference.py --prompt "The history of computing begins"

Interactive REPL:

python inference.py

Common options:

Flag	Default	Description
--prompt	(none)	One-shot continuation; omit for REPL
--temperature	0.8	Sampling temperature
--top-p	0.95	Nucleus sampling
--top-k	50	Top-k (0 disables)
--max-new-tokens	256	Generation length
--repetition-penalty	1.08	Repetition penalty
--context-length	2048	Tokens kept in the window
--seed	42	RNG seed
--device	cuda if available	cuda or cpu
--no-stream	off	Print the full completion at once
--no-bos	off	Do not prepend <|begin_of_text|>
--local-dir	(none)	Use a local snapshot directory

---

Files

kerzgrr/Haiku-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  chat_template.jinja
  tiny_gdn/
    __init__.py
    config.py
    model.py
    haiku_layers.py
    nn_common.py

---

Limitations

• Base model: not instruction-tuned; may ramble or fail at Q&A format
• Scale: ~655M parameters — research / edge prototype, not a frontier model
• Dependency: requires flash-linear-attention (KDA); not GGUF / llama.cpp compatible today
• Context: trained at 2,048; longer windows are experimental

---

Model family

Model	Parameters	Architecture	Stage	Hub
Monostich	~100M	LLaMA-style	SFT	kerzgrr/Monostich
Monostich-2-base	~150M	TinyGDN hybrid	Pretrain	kerzgrr/Monostich-2-base
Monostich-2	~150M	TinyGDN hybrid	SFT	kerzgrr/Monostich-2
Couplet-base	~268M	TinyGDN hybrid	Pretrain	kerzgrr/Couplet-base
Couplet	~268M	TinyGDN hybrid	SFT	kerzgrr/Couplet
Tercet-base	~502M	TinyGDN hybrid	Pretrain	kerzgrr/Tercet-base
Tercet	~502M	TinyGDN hybrid	SFT	kerzgrr/Tercet
Haiku-base	~655M	KDA + gated MLA	Pretrain	this repo

---

Citation

@misc{haikubase2026,
  title={Haiku-base: A 655M Hybrid KDA + Gated-MLA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Haiku-base}
}

---

Acknowledgments

• flash-linear-attention (Kimi Delta Attention)
• FineWeb-Edu
• Tercet family: kerzgrr/Tercet-base
• PyTorch SDPA / Hugging Face Hub + tokenizers

---

A haiku is three lines — larger than a tercet, still compact.