TounsiLM-8b

A Tunisian Arabic (Derja) conversational language model. LoRA adapter over CohereLabs/aya-expanse-8b, adapted in two stages: QLoRA continued pre-training on ~85M tokens of Tunisian Arabic text, then supervised fine-tuning on 31,669 instruction–response pairs with assistant-only loss.

Tunisian Arabic is the everyday language of over 12 million people, yet it is close to absent from LLM pre-training corpora. It is not a simplified Modern Standard Arabic: centuries of contact with French, Italian, Turkish, and Berber have left it distinct from MSA in phonology, morphology, vocabulary, and syntax, and distinct again from Moroccan and Egyptian Arabic. It also has no standardised orthography — speakers move freely between Arabic script and Latin-script Arabizi while code-switching with French and English mid-sentence. MSA-centric models degrade sharply under these conditions, and the failures are often invisible: a model produces fluent Arabic while silently substituting the wrong meaning.

A concrete example from our evaluation. Asked "kifeh nkhalli el ganaria ma tkahhalch?" ("how do I keep artichokes from turning dark?"), GPT-4o mapped the Tunisian word ganaria (artichoke) to its MSA homograph (canary) and returned advice about caring for small birds. Dialect adherence is not a stylistic nicety — it determines whether the model understood the question.

TounsiLM-8b was built as part of an end-to-end Tunisian Arabic spoken dialogue system (ASR → RAG → LLM) developed at INSAT, University of Carthage, in collaboration with Data2Innov.

Related artifacts

• alabenayed/improved-aya-expanse-8b-cpt-tunisian — the intermediate CPT checkpoint
• Syrinesmati/tunisian-dialect-corpus — CPT corpus
• Syrinesmati/tunisian-question-response-dataset — SFT dataset

---

Quick start

This is a PEFT adapter, not standalone weights. Load the base model, then apply the adapter.

The adapter already contains the continued-pre-training adaptation: SFT resumed training on the same LoRA parameters rather than stacking a second adapter, so you do not need to load the CPT checkpoint separately.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE = "CohereLabs/aya-expanse-8b"
ADAPTER = "alabenayed/TounsiLM-8b"

SYSTEM = (
    "أنت \"التيجاني\"، مساعد ذكاء اصطناعي تونسي. جاوب بالتونسي الدارجة فقط، "
    "وبالطول المناسب للسؤال: كان لازم قصير قصّر، وكان لازم شرح زيد أكثر. "
    "ممنوع الهلوسة أو الخروج على الموضوع."
)

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "أهلا، شنوة أخبارك اليوم؟"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=220,
    min_new_tokens=24,
    temperature=0.5,
    top_p=0.9,
    repetition_penalty=1.2,
    do_sample=True,
)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The generation defaults above are the ones used in the deployed dialogue system: a low temperature (0.5) to reduce drift toward MSA, and a repetition penalty of 1.2 to suppress the token-repetition artifacts that appear at longer outputs.

Requires roughly 18–20 GB of VRAM in bf16. For a single 16 GB GPU, load the base in 4-bit:

from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)

---

Why aya-expanse-8b

The base model was chosen by measurement, not default. Five open-weight candidates were compared on 80 hand-written Tunisian Arabic instructions (40 everyday assistant, 15 culture and food, 10 education and technology, 10 code-switched with French or English, 5 safety and refusal), producing 400 responses and 800 pairwise comparisons following a protocol adapted from TounsiBench (Ben Hassine et al., 2025). Quality, correctness, and relevance were scored as pairwise win rates; dialect adherence was scored independently per response on a 0–2 scale, because a response can be correct and relevant while sounding nothing like Derja.

Model	Quality	Correctness	Relevance	Tunisian usage (0–2)
Aya-Expanse-8B	0.64	0.68	0.71	0.80
Labess-7B	0.51	0.49	0.55	1.18
SILMA-9B	0.47	0.53	0.56	0.61
Llama-3-8B	0.42	0.45	0.48	0.44
Llama-3.2-1B	0.23	0.28	0.25	0.19

The interesting result is the split. Labess-7B produced the most authentically Tunisian output (1.18 vs 0.80) but its answers were consistently shorter and less informative. Aya-Expanse-8B won on instruction-following, correctness, and multilingual robustness — which matters for Tunisian code-switching across Arabic, French, and English — while its dialect adherence was merely adequate. Aya was selected on the reasoning that dialect can be taught by adaptation, whereas instruction-following capacity and multilingual grounding are much harder to add after the fact.

SILMA-9B, despite being Arabic-specialised, drifted persistently toward MSA phrasing — a direct illustration that strong Arabic performance does not transfer to Derja.

---

Training

Stage 1 — Continued pre-training (QLoRA)

Corpus. Aggregated from atakaboudi/Dialect_of_Tunisia-Work_Collection (69.4M tokens across 11 subsets including social media, ASR transcripts, sentiment corpora, and dialect-identification data) plus the Linagora Tunisian Derja collection (~1.76M lines across 14 subsets). Raw pool: ~2,057,757 samples, of which 93.7% Arabic script, ~6% mixed Arabic–Latin, and the remainder French or other, which were dropped.

Cleaning ran in four stages — conservative normalisation, script filtering, noise filtering, and two-pass deduplication with length control at the 99th percentile (370 words / 2,159 characters). Normalisation was deliberately restrained: dialectal spellings were not forced toward MSA equivalents, because over-normalising produces a corpus that no longer reflects real Derja. Mixed-script samples were retained where Arabic-script content remained dominant, to preserve authentic code-switching.

Final corpus: ~85M tokens, packed into 82,634 blocks of 1,024 tokens (84.6M tokens total, zero padding), split 81,808 train / 826 eval.

Objective	Causal language modelling
Epochs	1
Sequence length	1,024 tokens (concatenate-and-chunk packing)
Quantization	4-bit NF4, double quantization (bitsandbytes)
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning rate	2 × 10⁻⁴, cosine decay
Warmup	3% (~153 steps)
Optimizer	Paged AdamW 8-bit
Batch size	1 per device × 16 gradient accumulation = 16 effective
Total steps	5,113
Hardware	1 × NVIDIA GB10 (124 GB VRAM)
Duration	75 hours (3.53 × 10¹⁸ FLOPs)

Stage 2 — Supervised fine-tuning

Data. Started from 12,572 instruction–response pairs reviewed by native Tunisian speakers against three criteria: naturalness of the instruction in Derja, adequacy of the response, and realism of code-switching. Deduplication and MSA-dominant filtering reduced this to a 11,855-pair human-validated seed (−5.7%).

Controlled augmentation then expanded the seed by generating orthographic and code-switching variants via a 448-entry keyword-mapping dictionary — each frequently code-switched term mapped to both a phonetic Tunisian rendering and a native Arabic-script alternative (e.g. constipation → كونستيباسيون / إمساك; data science → داتا سيانس / علم البيانات) — plus conversational prefixes and parenthetical expansion. This produced 31,832 additional examples; after merging and final deduplication, 31,669 training-ready examples remained.

Read that composition carefully when interpreting the scale: the dataset is ~11.9K human-reviewed pairs plus mechanically derived variants, not 31.7K independently authored examples. The augmentation targets robustness to spelling and code-switching variation, which is the right goal for Derja, but it does not add 20K new semantic examples.

Split: 25,335 train / 6,334 test. Mean instruction length 10.2 words, mean response 36.1 words, across nine categories: advice_guidance (9,738), cultural_knowledge (6,429), general_knowledge (4,349), task_oriented (3,523), student_life, opinions_recommendations, emotional_support, everyday_conversation, storytelling.

Assistant-only loss. The tokenizer chat template was patched with a {% generation %} marker so TRL's SFTTrainer computes loss only over assistant tokens. The system prompt and user turn are conditioning context, not prediction targets — the objective is response quality, not reproducing the user's input.

Initialised from	alabenayed/improved-aya-expanse-8b-cpt-tunisian (adapter kept trainable)
Objective	Causal LM, assistant-only loss
Precision	bfloat16, no quantization
Epochs	2
Max sequence length	1,024 tokens
Learning rate	1 × 10⁻⁵, cosine decay
Warmup	3% (~47 steps)
Optimizer	AdamW fused (bf16), weight decay 0.01
Batch size	8 per device × 4 gradient accumulation = 32 effective
Total steps	1,582
Assistant tokens seen	~9.59 million
Hardware	1 × NVIDIA GB10 (124 GB VRAM)
Duration	14 hours (5.43 × 10¹⁷ FLOPs)

Framework versions

PEFT 0.19.1 · TRL 1.3.0 · Transformers 4.57.6 · PyTorch 2.11.0 · Datasets 4.8.5 · Tokenizers 0.22.2

---

Evaluation

Stage 1 — held-out language modelling loss

Measured on 826 packed blocks withheld from the CPT corpus.

Step	Train loss	Eval loss
25	3.633	—
600	2.402	2.367
1,800	2.236	2.210
3,400	2.130	2.115
5,113 (final)	2.105	2.089

A reduction of ~1.54 nats. The train–eval gap never exceeded 0.01 at any of the 26 evaluation passes, and gradient norms stayed within 0.38–0.47 throughout — the model generalised to unseen dialectal text rather than memorising the corpus.

Stage 2 — assistant-token metrics

Step	Epoch	Train loss	Token accuracy
10	0.01	3.602	42.2%
50	0.06	1.890	64.8%
200	0.25	1.240	73.5%
800	1.01	1.103	75.3%
1,582 (final)	2.00	1.062	76.2%

Total reduction of 2.54 nats (70.5%). The sharp early drop — 3.602 to 1.890 within 50 steps — indicates the CPT checkpoint already carried strong dialectal representations, leaving SFT to learn conversational structure rather than the language itself. Gradient norms held at 0.59–0.68 despite the transition from quantized CPT to full-precision bf16.

Downstream human evaluation

The most meaningful available evidence of output quality. 25 queries (15 in-domain to the cultural knowledge base, 10 out-of-domain), written across Arabic script, Arabizi, and French, rated 0–3 by a native Tunisian speaker blind to condition.

Condition	In-domain (n=15)	Out-of-domain (n=10)	Overall (n=25)
TounsiLM-8b alone	1.4	1.8	1.6
TounsiLM-8b + RAG	2.4	1.9	2.2

Bare, the model averages between "partially correct with major gaps" and "mostly correct." With retrieval grounding over a 1,647-entry curated knowledge base of Tunisian expressions, proverbs, food, rituals, and code-switching, in-domain quality gains a full point. The negligible out-of-domain gain (+0.1) is by design: confidence-gated injection suppresses retrieval below a mean RRF score of 0.60, so out-of-domain queries fall back to parametric generation.

If you are using this model for culturally specific queries, pair it with retrieval. The adaptation gave it Tunisian form; it did not give it reliable Tunisian facts.

---

Intended use

Conversational assistants, voice interfaces, and dialogue systems in Tunisian Arabic — in particular as the generation stage downstream of a Tunisian dialect ASR front-end, which is how it was built and deployed.

---

Citation

@misc{benayed2026tounsilm,
  title  = {TounsiLM-8b: A Tunisian Arabic Conversational Language Model},
  author = {Ben Ayed, Mohamed Ala and Smati, Syrine and Sassi, Yasmine},
  year   = {2026},
  note   = {End-of-year project, National Institute of Applied Science and
            Technology (INSAT), University of Carthage},
  url    = {https://huggingface.co/alabenayed/TounsiLM-8b}
}

Please also cite the base model and the benchmark that shaped the evaluation:

@article{dang2024ayaexpanse,
  title   = {Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier},
  author  = {Dang, John and Singh, Shivalika and D'Souza, Daniel and Ahmadian, Arash and
             Salamanca, Alejandro and Smith, Madeline and Peppin, Aidan and Hong, Sungjin and
             Govindassamy, Manoj and Zhao, Terrence and others},
  journal = {arXiv preprint arXiv:2412.04261},
  year    = {2024}
}

@inproceedings{benhassine2025tounsibench,
  title     = {TounsiBench: Benchmarking Large Language Models for Tunisian Arabic},
  author    = {Ben Hassine, Sarra and Arrak, Aymen and Addhoum, Mohamed and others},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural
               Language Processing},
  pages     = {34627--34642},
  year      = {2025}
}

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward
             and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif
             and Gallou{\'e}dec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}

---

Acknowledgements

Developed at INSAT, University of Carthage, supervised by Mrs. Hajer Taktak, in collaboration with Data2Innov. The CPT corpus draws on resources released by LINAGORA Research and by Ata Kabboudi.

Contact

Mohamed Ala Ben Ayed — alabenayed214@gmail.com · GitHub · LinkedIn