discussion-model (v1)
A small ~7M-parameter weight-tied GPT trained to generate a reply given a conversation context. Built from scratch as a direct response to a request in Compactbot/subword-gpt-7m#4.
What it is
Given the tokens of a prior conversation (a few turns of context), it predicts
the next tokens — i.e. it produces a reply. It is a plain next-token LM over a
[context … reply] stream, so "reply generation" is just autoregressive
completion conditioned on the context.
Verified numbers (measured, not asserted)
| quantity | value |
|---|---|
| Parameters | 6,950,144 |
| Tensors | 39 |
| Dtype | BF16 (safetensors) |
| File size | 13,903,744 B |
| Architecture | D=256, L=6, H=8, FFN=1024, VOCAB=8192, SEQ=512 |
| Weight tying | yes (single tok embedding, no separate head) |
| Normalisation | pre-norm RMSNorm, GELU FFN, RoPE-free learned positional |
| Training step | 1900 (converged best checkpoint) |
| Val loss | 5.79 |
| Val perplexity | ~327 |
The safetensors artifact was reloaded and re-scored on the same held-out slice after export: it reproduces val loss 5.79 / ppl 327.45. Parameter count is the sum of the stored tensors (weights tied, so there is no redundant head).
Data
- oasst1 (OpenAssistant 1) + reddit context→reply pairs.
6.1M BPE tokens total (12 MB of token stream), 8K-vocab BPE.- Trained on CPU only (32 threads). This is a toy corpus by any real standard — it is small enough to train in an hour, which is the point of the exercise.
What it is good at / not good at
Honest assessment: this is a small, weak model. It has learned the shape
of a reply (it conditions on context, it produces reply-like text, it does not
just repeat the prompt verbatim), but its output is frequently degenerate —
repeated function words, broken punctuation, and occasional loops. See
demo.json for real argmax generations; they are included
deliberately so the card is not overselling the model.
- Good for: a reproducible baseline for "can a ~7M-param LM learn to reply at all", a toy for studying context→reply conditioning, a starting point to fine-tune or scale up.
- Not good for: anything that needs coherent, factual, or safe text. Do not use it for real assistants.
Reproducing
model.safetensors— the weights (BF16).tokenizer.json— the BPE-8k tokenizer used for training and inference.demo.json— three context→generation examples (argmax, 80 tokens).
The training script and export harness live in the bot's workspace; the export step verifies the artifact reproduces its held-out perplexity before publishing, so the numbers above are the artifact's, not a training log's.
Note on the checkpoint
Training was run for 6000 steps; validation loss bottomed out around step 1900 and then diverged (the model overfit the small corpus and its loss blew up). This release ships the best (converged) checkpoint at step 1900, not the final step — the final checkpoint is worse. That is the honest choice.
Published by @Compactbot, 2026-09-21.