C1 -- base sinusoidal MHA LayerNorm subword
Encoder-decoder transformer written from scratch -- no nn.Transformer, no
pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary
cipher into English plaintext. One row of a five-configuration ablation in
which exactly one architectural choice moves at a time.
| configuration | C1 -- base sinusoidal MHA LayerNorm subword |
| positional encoding | sinusoidal |
| attention | mha |
| normalisation | layernorm |
| tokenisation | unigram LM subword, fitted on the training split |
| depth | 2 encoder / 2 decoder layers |
| width | dim_model 128, 4 heads, dim_ff 512 |
| parameters | 2,763,264 |
| windowing | 1024-character source windows at stride 768 (25,654 training windows) |
| best epoch | 98 (val loss 0.0942) |
Test metrics
| metric | test | test_full | gen_test_win | gen_test | gen_test_full |
|---|---|---|---|---|---|
| loss | 0.1108 | 7.0169 | -- | -- | -- |
| perplexity | 1.1172 | 1115.3036 | -- | -- | -- |
| token acc | 0.9776 | 0.2704 | 0.8477 | 0.4725 | 0.1503 |
| bit acc | -- | -- | 0.9586 | 0.8195 | 0.2315 |
| seq acc | 0.5972 | 0.0482 | 0.7002 | 0.2520 | 0.0520 |
| levenshtein | -- | -- | 0.9238 | 18.9620 | 472.0240 |
| lev / len | -- | -- | 0.0075 | 0.0283 | 0.6332 |
| BLEU | -- | -- | 0.9566 | 0.9368 | 0.0575 |
| ROUGE-1 | -- | -- | 0.9800 | 0.9692 | 0.4304 |
| ROUGE-2 | -- | -- | 0.9636 | 0.9519 | 0.3962 |
| ROUGE-L | -- | -- | 0.9800 | 0.9691 | 0.4292 |
test-- teacher-forced, per windowtest_full-- teacher-forced, per whole linegen_test_win-- greedy, per windowgen_test-- greedy, windowed and stitchedgen_test_full-- greedy, whole line in one pass
Files
C1_best.pt-- weights, optimiser state, config and vocabulary sizes, from the best epoch.config.json-- the full ExperimentConfig this was trained under.summary.json-- the run record: the metrics above plus per-epoch history.tokenizer_cipher.json,tokenizer_plain.json-- the fitted unigram vocabularies, when the config uses them. Byte-level configurations have none by construction.
Loading it
The checkpoint is a plain torch.save dict, not a transformers model. The
vocabulary sizes travel with it because they are not recoverable from the
config -- the tokenizer lands wherever pruning leaves it:
import torch
from huggingface_hub import hf_hub_download
from src.config import get_config
from src.models.base import build_model
path = hf_hub_download("winterdewdev/anlp-a1-transformers-C1", "C1_best.pt")
state = torch.load(path, map_location="cpu", weights_only=False)
cfg = get_config(state["config_name"])
model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"])
model.load_state_dict(state["model"])
model.eval()
Trained with python -m src.train --config C1.
- Downloads last month
- 11