GTM-v2-chat by OpenGCM

A ~120M parameter, decoder-only GPT-style model, base-pretrained from scratch and then fine-tuned (SFT) on UltraChat to behave like a chat assistant. Single RTX Pro 6000, ~4.16B pretraining tokens + ~1B SFT tokens.

This is the SFT/chat variant of GTM-v2-base — same architecture and pretrained weights, continued with supervised fine-tuning on conversational data. This is still a small, from-scratch, single-GPU hobby/research model, not a production assistant. It is far less capable and less reliable than commercial chat models, and can confidently produce fluent but incorrect answers.

What SFT changed

The base model (GTM-v2-base) does not answer questions — given "What is the capital of France?" it continues the text as if it were part of a document, without ever reliably stating the answer, no matter how it's prompted. After SFT on UltraChat, the same underlying model answers directly:

<|user|>
What is the capital of France?
<|assistant|>
The capital of France is Paris. The city has seven major cities including
London, Paris, and Paris itself.<|endofturn|>

That's a real, qualitative capability shift — the model went from never producing a correct, direct answer across an entire base-pretraining run to answering directly, with the correct fact, after SFT. This is one example, not a rigorous evaluation — it is not yet known how often this succeeds versus fails on other questions. Base-model benchmark scores (HellaSwag, ARC) were not meaningfully different pre- vs. post-SFT, since those are likelihood-scored on fixed text and don't test chat-style question answering.

Chat template

The tokenizer has no special chat tokens — conversations are plain text with these literal markers:

<|user|>
{user turn}
<|assistant|>
{assistant turn}<|endofturn|>

Prompt the model by ending your input with <|assistant|>\n so it continues as the assistant turn.

Model details

• Architecture: nanoGPT-style decoder-only transformer (same as GTM-v2-base) — 14 layers, 8 attention heads, 704 embedding dim, ~119.4M parameters, 1024 token context, PyTorch fused scaled_dot_product_attention.
• Tokenizer: tiktoken GPT-2 BPE encoding (tiktoken.get_encoding("gpt2")), vocab size 50,257.
• Optimizer: Muon (2D weight matrices) + AdamW (embeddings, layernorms, biases), fresh optimizer state for the SFT stage (not carried over from pretraining).
• Precision: trained with bf16 autocast; released weights are fp32.
• Base pretraining: ~4.16B tokens (see GTM-v2-base for the full data mix: FineWeb-Edu, Cosmopedia-v2, FineMath, FineWeb).
• SFT data: UltraChat (train_sft split), ~266M actual tokens after tokenization — smaller than UltraChat's advertised size once encoded. Reaching the ~1B token SFT target meant roughly 3.8 epochs over this data.
• SFT loss masking: loss is computed only on assistant-turn tokens; user turns and template markers are masked out of the loss (standard SFT practice), so the model is trained to generate good responses, not to predict user text.
• SFT learning rate: notably lower peak LR than base pretraining (5e-5 vs. 3e-4), standard practice for fine-tuning on top of a pretrained model.

Known limitations

• Does not reliably stop at its own <|endofturn|> marker. The generation loop's early-stopping is keyed to the tokenizer's built-in end-of-text token, not the custom <|endofturn|> text marker learned during SFT. In practice this means the model often continues generating past a complete, correct answer into unrelated rambling (as in the example above, where a correct answer is followed by unrelated, factually shakier text about Paris as a "port city"). If integrating this model, truncate generated text at the first <|endofturn|> occurrence yourself rather than relying on the model to stop cleanly.
• Correctness is not guaranteed even when it answers directly. SFT taught response format (answer the question, use the assistant turn structure) more reliably than it taught factual accuracy. Expect confidently wrong answers on many prompts.
• Only lightly evaluated. The example above is a single, deliberately chosen (cherry-picked) success case shared to illustrate the format shift SFT produced, not a claim about typical response quality. No systematic chat-quality evaluation (multiple prompts, failure-rate measurement, comparison against other small chat models) has been done yet.
• 3.8 epochs over a relatively small SFT set (266M tokens) raises some risk of overfitting to UltraChat's specific style/topics rather than generalizing broadly across conversation types.
• Inherits GTM-v2-base's limitations where SFT didn't specifically address them: no code capability (no code data in either pretraining or SFT), limited context (1024 tokens), small overall knowledge base relative to models trained on much larger corpora.

Usage

Requires model.py (included in this repo) alongside the checkpoint — this is a plain PyTorch model, not a transformers AutoModel.

pip install torch safetensors tiktoken

import json
import torch
from safetensors.torch import load_file
from model import GPT, GPTConfig

with open("config.json") as f:
    cfg_dict = json.load(f)
config = GPTConfig(
    vocab_size=cfg_dict["vocab_size"], block_size=cfg_dict["block_size"],
    n_layer=cfg_dict["n_layer"], n_head=cfg_dict["n_head"],
    n_embd=cfg_dict["n_embd"], dropout=cfg_dict["dropout"], bias=cfg_dict["bias"],
)
model = GPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

import tiktoken
enc = tiktoken.get_encoding("gpt2")

prompt = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"
ids = enc.encode_ordinary(prompt)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    out = model.generate(
        x, max_new_tokens=128, temperature=0.8, top_k=50,
        eot_token=enc.eot_token, repetition_penalty=1.3,
    )

response = enc.decode(out[0].tolist())
# manually truncate at the end-of-turn marker -- the model does not
# reliably stop generating here on its own (see Known limitations)
response = response.split("<|endofturn|>")[0] + "<|endofturn|>"
print(response)

License

Apache 2.0 for this repo's contents (model weights, model.py, and this README). Base-pretraining data retains its own license (FineWeb, FineWeb-Edu, Cosmopedia-v2, FineMath: ODC-BY-1.0). UltraChat (HuggingFaceH4/ultrachat_200k) is MIT licensed. This repo distributes model weights, not the training data itself.