VenyaVel/anlp-a1-transformers-blt

ANLP Assignment 1 — Transformers from Scratch, Architectural Ablation, and BLT

Custom Encoder-Decoder Transformer checkpoints for a controlled 5-config architectural ablation study, trained from scratch (no nn.Transformer / nn.MultiheadAttention) on an encrypted-binary-to-plaintext sequence-to-sequence task. Task: IIIT-H Advanced NLP, Assignment 1.

These are raw PyTorch state_dict checkpoints for a custom architecture. This repo does not include the model source code (submitted separately as part of the assignment zip, not published here). To load a checkpoint, copy src/models/{attention,positional,norm,transformer,blt}.py from the assignment code and:

import torch
from src.models.transformer import Seq2SeqTransformer
from src.models.blt import BLTSeq2SeqTransformer

# C1-C4 (subword BPE, encoder-decoder transformer)
model = Seq2SeqTransformer(
    src_vocab_size=4096, tgt_vocab_size=2048,   # from cipher_bpe_tokenizer.json / plain_bpe_tokenizer.json
    dim=512, num_layers=4, num_heads=16, ff_dim=1024,
    max_seq_len=200, dropout=0.1,
    pe_type="sinusoidal", attention_type="mha", norm_type="layernorm",  # see table below per config
)
model.load_state_dict(torch.load("c1_checkpoint.pt", map_location="cpu"))
model.eval()

# C5 (BLT, token-free byte-level)
blt = BLTSeq2SeqTransformer(
    vocab_size=259, byte_dim=512, patch_dim=512, patch_size=4,
    num_layers=4, num_heads=16, num_kv_heads=16, ff_dim=1024, max_seq_len=200,
    pe_type="sinusoidal", attention_type="mha", norm_type="layernorm", dropout=0.1,
)
blt.load_state_dict(torch.load("c5_checkpoint.pt", map_location="cpu"))
blt.eval()

Task

Map an encrypted binary sequence (0/1 string, 1 byte per plaintext character, verified 1:1 aligned) to its plaintext English sentence. Trained on 5,000 line-aligned pairs (Brown corpus based), windowed into 1024-bit (128-byte/char) non-overlapping chunks per line for training, with train/val/test split (90/5/5, seed 42) done at the line level before windowing to prevent leakage.

Configurations

Each config changes exactly one component from the C1 baseline (dim=512, layers=4, heads=16, ff_dim=1024, dropout=0.1, label_smoothing=0.05, lr=3e-4, batch_size=64, up to 200 epochs with early stopping patience=50):

Config	Positional Encoding	Attention	Normalization	Tokenization
C1 (baseline)	Sinusoidal	Multi-Head Attention	LayerNorm	From-scratch subword BPE
C2	RoPE	Multi-Head Attention	LayerNorm	From-scratch subword BPE
C3	Sinusoidal	Grouped-Query Attention (4 KV heads)	LayerNorm	From-scratch subword BPE
C4	Sinusoidal	Multi-Head Attention	RMSNorm	From-scratch subword BPE
C5	Sinusoidal	Multi-Head Attention	LayerNorm	BLT (token-free, byte-level, fixed patch_size=4)

Both tokenizers (cipher_bpe_tokenizer.json, plain_bpe_tokenizer.json) are included in this repo — from-scratch byte-level / word-level BPE, no tokenizers/sentencepiece library used to train them.

Test set results

Greedy decoding only. Whole-sentence (reassembled) is the number comparable to "Sequence Accuracy" as literally defined in the assignment spec; per-window is a secondary, easier metric (see assignment README for why both are reported).

Whole-sentence (n=250 lines)

Config	Bit-Level Acc	Sequence Acc	Levenshtein	BLEU	ROUGE-L
C1	0.8498	0.216	2.216	0.9439	0.9762
C2	0.8515	0.216	2.576	0.9441	0.9756
C3	0.8520	0.212	2.384	0.9425	0.9753
C4	0.8438	0.208	2.304	0.9434	0.9755
C5	1.0000	1.000	0.000	1.0000	1.0000

Per-window (n=1308 windows)

Config	Bit-Level Acc	Sequence Acc	Levenshtein	BLEU	ROUGE-L
C1	0.9909	0.9144	0.162	0.9712	0.9951
C2	0.9926	0.9098	0.231	0.9707	0.9949
C3	0.9907	0.8998	0.194	0.9694	0.9944
C4	0.9879	0.9014	0.179	0.9698	0.9934
C5	1.0000	1.0000	0.000	0.9826	1.0000

Training logs

Full training curves and per-run system metrics (peak GPU memory, wall-clock time) are logged on Weights & Biases: velven1206-iiit-hyderabad/ANLP A1 Windowed v3.