gemma-3n-bm-base

A LoRA adapter for Gemma 3n E4B — the pretrained, non-instruction-tuned variant — trained for Bambara with Unsloth and TRL.

Adapter weights only. Load onto unsloth/gemma-3n-E4B-unsloth-bnb-4bit; that checkpoint is 4-bit NF4, and the adapter should be loaded onto the same quantised base it was trained against.

Config

Rank r	128
lora_alpha	32
use_rslora	true — effective scale is alpha / sqrt(r) ≈ 2.83, not alpha / r
lora_dropout	0

Adapted: the attention and MLP projections of the 35 decoder layers, the audio tower's attention projections, and — unusually for a LoRA — embed_tokens and lm_head, which is what makes this the vocabulary-adaptation stage.

Usage

import torch
from transformers import AutoProcessor, Gemma3nForConditionalGeneration
from peft import PeftModel

base = Gemma3nForConditionalGeneration.from_pretrained(
    "unsloth/gemma-3n-E4B-unsloth-bnb-4bit",
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(base, "djelia/gemma-3n-bm-base")
model.eval()

processor = AutoProcessor.from_pretrained("djelia/gemma-3n-bm-base", padding_side="left")

inputs = processor(text="Bamako ye ", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(processor.decode(out[0], skip_special_tokens=True))

Notes

This adapter sits on the pretrained base, so use plain text continuation rather than chat formatting.

bitsandbytes and accelerate are required for the 4-bit base. On transformers releases older than the dtype= rename, pass torch_dtype=torch.bfloat16.