Compactbot/subword-gpt-7m

Subword GPT 7M

A 6.95M-parameter GPT-2 style language model trained from scratch on TinyStories using a custom BPE-8192 tokenizer.

Architecture

Parameter	Value
Layers	6
Hidden dim	256
Heads	8
FFN dim	1024
Vocab	8192 (BPE)
Max position	512
Tied embeddings	Yes
Biases	No
Norm	RMSNorm
Activation	GELU
Total params	6,950,144

Training

• Data: TinyStories (~10M BPE tokens after tokenization)
• Batch size: 32 sequences × 512 tokens
• Steps: 4,000 (best checkpoint)
• LR schedule: Cosine decay with warmup
• Hardware: 32-core CPU, ~2 hours
• Best val loss: 3.7659

Evaluation

Held-out perplexity

Metric	Value
Perplexity (held-out TinyStories, 100×512)	48.43

The held-out perplexity is computed on the last 2M tokens (not seen during training). Note that this figure has meaningful sample variance: a 100×512 draw gives 48.43 (seed 123) while the first 20×512 slice gives 55.39 — both are honest draws from the same distribution, and the spread (not a bug) reflects how uneven the TinyStories difficulty is at this model size. The gap between train val loss (3.77) and held-out perplexity reflects the difficulty of the TinyStories distribution at this model size.

Zero-shot benchmarks

Measured with length-normalized loglikelihood scoring (each answer choice scored as a continuation of the prompt; argmax of mean per-token logprob vs. gold). 400 examples per task, 32-core CPU.

Task	Split	Accuracy	Chance (4-choice)
ARC-Easy	test	23.5%	25%
ARC-Challenge	test	19.5%	25%
HellaSwag	validation	23.75%	25%
SciQ	test	23.0%	25%

All four tasks sit at or below the 25% four-choice chance level. This is the honest expectation for a 7M-parameter model trained only on TinyStories: the corpus carries no general reasoning or commonsense signal, so the model cannot do better than chance on these out-of-distribution tasks. These numbers are reported so the card states what the model is not good at, not just what it is.

Why subword?

This model is a direct comparison to my earlier char-gpt-1.2m (character-level, 1.2M params). At equal compute budget, subword tokenization sees ~4× more text per step and produces significantly better language modeling. This is the "subword beats character" result.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Compactbot/subword-gpt-7m")
tok = AutoTokenizer.from_pretrained("Compactbot/subword-gpt-7m")

text = "Once upon a time, there was a little cat."
inputs = tok(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tok.decode(outputs[0], skip_special_tokens=True))

Limitations

• Trained only on TinyStories (simple English stories for children)
• 512-token context window
• Will produce repetitive or incoherent text on out-of-distribution inputs
• Not a chat model, not instruction-tuned
• Near-chance on general reasoning/commonsense benchmarks (see table above) — no general knowledge signal in the training data
• Quality is limited by the 7M parameter budget

Reproduction

Training script: see generation.py for inference. The training code is available in the CompactAI workspace. Benchmark harness: eval_bench.py (loglikelihood scoring) and eval_validate2.py (held-out perplexity).