GitHub | Model Download | Dataset Download | Inference Space |
A Compact Bidirectional Encoder-Decoder Transformer-Based Model for English-Khmer Translation
1. Abstract
This repository presents Netra-NMT, a compact 77M-parameter encoder-decoder transformer-based model trained from scratch on 220 million tokens of English-Khmer parallel text (4.2M bidirectional examples). The encoder uses bidirectional self-attention, much like BERT, to capture global contextual representation. The decoder performs autoregressive generation through causal self-attention and encoder-decoder cross-attention.
Netra-NMT adopts a modern transformer recipe: rotary position embeddings (RoPE) in self-attention (no learned position table), RMSNorm with Pre-Normalization for stable optimization, SwiGLU feed-forward networks, and a single embedding table shared across the encoder input, decoder input, and tied output projection. It uses a deep-encoder / shallow-decoder layout (12 encoder layers, 2 decoder layers) which β combined with a KV cache for O(T) incremental decoding β delivers autoregressive quality at a fraction of the decoding latency (Kasai et al., 2021). Khmer text is word-segmented with khmercut before tokenization, so the shared 32K SentencePiece vocabulary sees real word boundaries.
2. Dataset
Netra-NMT was trained on 220 million tokens drawn from approximately 2.4 million unique English-Khmer sentence pairs (4.2 million examples after bidirectional augmentation). The corpus combines LLM-generated synthetic data with web-crawled parallel text, spanning legal, literary, medical, technical, and conversational domains.
2.1 Sources
| Dataset | Type | Pairs | Domains |
|---|---|---|---|
| Darayut/khmer-english-pairs-raw | Synthetic | 200K | Legal, Literary, Governmental |
| lyfeyvutha/nllb-en-km-316K | Synthetic | 316K | General |
| KrorngAI/ParaCrawl-English-Khmer-v2 | Web crawl (ParaCrawl) | 1.5M | Web / general |
| SeyhaLite/Translate-English-Khmer-All | --- | 366K | General |
| Total | 2.4M |
2.2 Preprocessing
Raw data was cleaned through the following pipeline:
- Deduplication: exact duplicate pairs removed across all sources.
- Length filtering: pairs with extreme source/target length mismatches were discarded.
- Empty/null removal: pairs where either side was empty or below a minimum token count were dropped.
- English source case-normalization: English is NFC-normalized, whitespace-collapsed, and lowercased only when it is the
en2kmsource, so"I love Cambodia"and"i love cambodia"map to identical model inputs. English kept as akm2entarget retains its natural casing. - Khmer word segmentation: Khmer (which is written without spaces) is segmented with khmercut wherever it appears β as the
en2kmtarget and thekm2ensource β matching the segmented tokenizer. Khmer output is de-segmented back to natural text at inference time.
After cleaning, each surviving pair is duplicated in both directions (ENβKM and KMβEN) with a direction prefix token (<2km> / <2en>), yielding ~4.2 million training examples.
3. Model Architecture
Figure 1: Overview of the Netra-NMT encoder-decoder architecture. The encoder (left) processes the source sentence with bidirectional self-attention; the decoder (right) generates the target sentence autoregressively via causal self-attention and cross-attention over the encoder output. Both sides share a 32K SentencePiece tokenizer.
Netra-NMT follows an encoder-decoder transformer architecture modernized for training stability, parameter efficiency, and low-latency decoding.
Encoder takes the source sentence tokenized by the shared 32K SentencePiece tokenizer (Khmer is word-segmented with khmercut beforehand), and passes the sequence through 12 transformer layers with bidirectional self-attention (every token attends to every other token, similar to BERT). Positions are encoded with rotary embeddings (RoPE) applied to the queries and keys inside self-attention β there is no learned position table. A final RMSNorm is applied to the encoder output before it is passed to the decoder via cross-attention.
Decoder takes the (partially generated) target sentence through the same tokenizer and passes it through only 2 transformer layers. Each decoder layer applies three sub-layers in order: (1) causal (masked) self-attention with RoPE over previously generated tokens, backed by a KV cache for O(T) incremental decoding, (2) cross-attention over the full encoder output, and (3) a SwiGLU feed-forward block. A final RMSNorm feeds into the tied projection head. The deep-encoder / shallow-decoder split (12 vs 2) follows Kasai et al. (2021): most of the modeling capacity lives in the parallelizable encoder, while the thin decoder β the part that runs sequentially at inference β keeps per-step cost low without sacrificing quality.
Architectural improvements over the vanilla transformer:
| Feature | Detail |
|---|---|
| Rotary Position Embeddings (RoPE) | Positions encoded by rotating Q/K inside self-attention β no learned position table, and generalizes past the training length |
| RMSNorm (Pre-Norm) | Root-mean-square normalization applied before each sub-layer; as stable as LayerNorm and cheaper |
| SwiGLU FFN | Feed-forward blocks use the SwiGLU activation instead of ReLU, providing richer representational capacity |
| Shared embeddings | One embedding table is shared across the encoder input, decoder input, and the tied output projection head |
| Deep encoder / shallow decoder | 12-layer encoder + 2-layer decoder (Kasai et al., 2021) β retains quality while sharply cutting sequential decoding cost |
| KV cache | Incremental decoding caches past keys/values, making generation O(T) instead of O(TΒ²) |
| Khmer word segmentation | Khmer is segmented with khmercut before SentencePiece so the tokenizer learns real word boundaries |
Hyperparameters:
| d_model | 512 |
| Encoder / Decoder layers | 12 / 2 |
| Attention heads | 8 |
| FFN hidden size | 2048 |
| Position encoding | Rotary (RoPE) |
| Normalization | RMSNorm (Pre-Norm) |
| Vocabulary | 32K (SentencePiece unigram, shared, Khmer-segmented) |
| Total parameters | ~77M |
4. Evaluation Results
Install
pip install netra-nmt # core (Python API + CLI)
pip install "netra-nmt[web]" # + FastAPI web app & REST API
Or from source:
git clone https://github.com/NDarayut/netra-nmt
cd netra-nmt
pip install -e ".[web]"
The first translation downloads the weights (180 MB fp16) from the Hugging Face Hub and caches them
under `/.cache/huggingface`.
Usage
1. Python API
from netra_nmt import NetraTranslator
t = NetraTranslator() # auto-detect GPU/CPU; downloads weights once
t.translate("Hello, how are you?", direction="en2km") # β "αα½ααααΈαα»ααααααΆαα’αα?"
t.translate("αααα»ααααα‘αΆαααααααααααααααα»αα", direction="km2en")
# Batch + decoding options
t.translate_batch(["Good morning.", "See you tomorrow."], direction="en2km")
t.translate("Good morning, my friend.", direction="en2km", mode="beam", beam_size=5)
One-shot helper (caches a default translator):
from netra_nmt import translate
translate("Hello", direction="en2km")
direction is "en2km" (EnglishβKhmer) or "km2en" (KhmerβEnglish).
mode is "greedy" (default), "beam", or "sample".
2. CLI
# Single sentence (default direction en2km):
netra-translate --text "Hello, how are you?"
# Khmer β English with beam search:
netra-translate --text "αα½ααααΈ, ααΎα’ααααα»ααααααΆααα?" --direction km2en --mode beam
# Translate a file (one sentence per line):
netra-translate --file input.txt --output output.txt --direction en2km
# Interactive REPL (omit --text / --file):
netra-translate
3. Web app + REST API (FastAPI)
netra-web # serves the web UI + API at http://127.0.0.1:8000
netra-web --port 8080 --device cpu
netra-web --local-dir export # load weights from a local export dir
A two-pane translation site (source left, output right, ENβKM swap button) and a JSON API:
curl -X POST http://127.0.0.1:8000/api/translate \
-H 'Content-Type: application/json' \
-d '{"text": "Hello, how are you?", "direction": "en2km"}'
# {"translation": "...", "direction": "en2km"}
Requires the web extra (pip install "netra-nmt[web]").
- Downloads last month
- 158