Compactbot/char-gpt-1.2m

Char-GPT 1.2M

A tiny character-level causal transformer trained from scratch on TinyStories. A small, honest reference build — the point is a model whose card matches its artifact exactly, not a competitive checkpoint.

Parameters (exact)

1,216,000 parameters, untied head.

module	params
transformer.wte (65×128)	8,320
transformer.wpe (128×128)	16,384
6 × attention (qkv + proj, bias-free)	614,400
6 × FFN (4×, bias-free)	552,960
6 × 2 LayerNorm (affine)	1,536
ln_f (128)	256
lm_head (65×128, separate / untied)	8,320
total	1,216,000

The head is not weight-tied: the checkpoint stores two distinct 65×128 tensors (transformer.wte.weight and lm_head.weight), and model.py never assigns one to the other. config.json therefore says tie_word_embeddings: false. (If the head were tied the count would be 1,207,680.)

Architecture

nanoGPT-style GPT-2, all bias-free except LayerNorm:

• n_layer=6, n_head=4, n_embd=128, FFN = 4× = 512
• vocab_size=65 (printable ASCII + newline), block_size=128
• RoPE: none (learned positional embedding wpe)

Training

• Data: roneneldan/TinyStories (train split), first ~1.0M characters, 90/5/5 train/val/test split by character.
• Steps: 1,500, batch 32 × seq 128, AdamW (lr 6e-4, cosine, warmup), grad-clip 1.0, float32, CPU (16 threads). ~3.5 min.
• Seed: 42.

Quality — what it is and is not

Held-out perplexities (measured on the full held-out test split, 2026-09-20):

split	loss	perplexity
test (49,674 tokens)	1.4369	4.21

It captures TinyStories' surface style (short declarative sentences, simple vocabulary, character names) but it is a 1.2M-parameter model on ~1M characters — it does not grasp meaning, it repeats and drifts, and it will produce the kind of plausible-looking-but-nonsense text in sample.txt. Treat it as a working toy / reference architecture, not a useful language model.

The original training log reported val 1.9046 / test 1.9473 from a 60-batch random evaluation; the full-split number above is the honest one.

Files

• model.safetensors — 4,869,112 B (53 tensors, F32)
• model.py — CharGPT + from_config
• config.json, tokenizer_config.json (char vocab)
• sample.txt — 240-char greedy-ish sample
• LICENSE — Apache-2.0

Reproduce

import torch, json
from model import from_config
cfg = json.load(open("config.json"))
m = from_config(cfg)
print(sum(p.numel() for p in m.parameters()))  # 1216000