Alo-50M-Base-Retied

Model Summary

Alo-50M-Base-Retied is a compact, 50-million parameter foundational Large Language Model (LLM) built exclusively for the Bengali (Bangla) language. Trained entirely from scratch, it aims to lower the compute barrier for Bengali NLP research and provide a viable path for deploying localized AI assistants on standard CPUs and resource-constrained edge devices.

⚠️ Note on Weight Tying (Update): This model is the updated, mathematically tied version of the original Alo-70M-Base. By correctly tying the input and output embeddings (tie_word_embeddings = True), we eliminated a duplicate copy of the $16.8\text{M}$ parameter embedding matrix from the checkpoint. This successfully reduced the active parameter footprint from 69M to 52M and shrank the disk/download size by 24% (from 276MB to 209MB) while maintaining mathematically identical outputs and evaluation performance.

• Developer: Fahad Hossain
• Language: Bengali (Bangla)
• Model Type: Causal Language Model (Autoregressive Transformer)
• Parameter Count: 52 Million (Active/Tied) / Formerly 69 Million (Untied)
• License: Apache 2.0

Related Resources

This model is the base foundational model. We also release the instruction-tuned version, the synthetic dataset used for alignment, and the standalone tokenizer:

• Instruction-Tuned Model: spitfire4794/Alo-70M
• Alignment Dataset: spitfire4794/Bangla-SFT-50k
• Tokenizer: spitfire4794/beng_bpe

Usage

You can load and generate text with Alo-50M-Base-Retied using the transformers library.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70m-Base-retied"
tokenizer_id = "spitfire4794/beng_bpe"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "বাংলাদেশের রাজধানী ঢাকা"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=50, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Details

Alo-50M-Base-Retied is based on a scaled-down LLaMA transformer architecture optimized for hardware alignment and parameter efficiency.

• Layers: 12
• Hidden Dimension ($d_{model}$): 512
• Intermediate FFN Dimension: 1408
• Attention Mechanism: Grouped-Query Attention (GQA) with 8 query heads and 4 key-value heads.
• Positional Embeddings: Rotary Position Embeddings (RoPE) with a base frequency theta of 10,000.
• Activation Function: SwiGLU (silu)
• Word Embeddings: Tied (tie_word_embeddings = True). The input embeddings (embed_tokens) and output language model head (lm_head) share the exact same weight matrix. This resolves a prior configuration issue where identical matrices were loaded twice into GPU memory, reducing the static lookup table footprint from 33.5M parameters down to 16.8M parameters.
• Context Window: 1024 tokens.
• Tokenizer: Custom Bengali Byte-Pair Encoding (BPE) tokenizer (spitfire4794/beng_bpe) with a highly optimized vocabulary size of 32,768 (2^15) designed to maximize TPU efficiency.

Pre-Training Data

The model was pre-trained on a curated, strictly Bengali corpus containing approximately 19.25 billion tokens. The dataset is a blend of:

1. FineWeb-2 (Bengali Split): Providing diverse, web-scale filtered documents.
2. TituLM Bengali Corpus (CommonCrawl Split): Introducing balanced, general-domain web data.

Training Procedure

Alo-50M-Base-Retied (initially trained as Alo-70M-Base) was initialized with completely randomized weights (no transfer learning) and trained from scratch.

• Hardware: Kaggle Tensor Processing Unit (TPU) v5e-8
• Training Time: 12.9 hours (wall time) across two sequential runs.
• Optimizer: AdamW (adam_torch_xla) with beta_1 = 0.9, beta_2 = 0.95, epsilon = 10^-8
• Weight Decay: 0.1
• Learning Rate Schedule: Cosine Decay with a gradual warmup phase, peaking at 6 times 10^-4.
• Precision & Optimization: Mixed bfloat16 (bf16) precision with gradient checkpointing enabled to maximize throughput.

Evaluation Results

The model was evaluated in a zero-shot setting across various Bengali reasoning and knowledge benchmarks using a continuation-based evaluation methodology (calculating conditional log-probabilities directly over raw Bengali text to avoid "pointer-reasoning" bottlenecks).

The evaluation metrics below represent the actual raw benchmark results computed directly from the tied checkpoint.

Benchmark	Alo-50M-Base-Retied	Gemma-3-270M-IT	TigerLLM-1B-IT
bangla_mmlu_bn	26.30%	26.81%	27.66%
bangla_commonsenseqa_bn	28.58%	22.77%	25.14%
indicbench_arc_bn_challenge	23.04%	25.34%	27.13%
boolqa_bn	48.97%	51.30%	52.40%
openbookqa_bn	30.78%	31.99%	34.21%
piqa_bn	50.49%	49.51%	49.51%
hellaswag_bn	27.17%	27.85%	31.01%

Note: Alo-50M-Base-Retied matches or outperforms both the 270M and 1B baselines on CommonsenseQA and PIQA despite having 5x to 19x fewer active parameters.

Limitations and Biases

• Encyclopedic Knowledge: Due to the severe parameter constraints (~50M parameters), the model physically lacks the parametric capacity to act as a deep encyclopedic knowledge base, which reflects in its performance on knowledge-intensive retrieval tasks like MMLU and ARC-Challenge.
• Tokenizer Evaluation Bias: During continuation-based log-likelihood evaluations, Alo-50M's highly efficient native tokenizer may possess a mathematical advantage over multilingual baselines (which heavily fragment Bengali text), contributing partially to its high performance margins on specific tasks.
• Instruction Following: This is a base model. It will attempt to complete text rather than follow conversational instructions. For chat and instruction-following, please use Alo-70M.

Citation

Technical paper out soon.