Sophira-1B-base

Model Summary

• Model name: Sophira-1B-base
• Intended Hugging Face repository: Gpeik/Sophira-1B-base
• Internal frozen release candidate: Sophira-1B-base-v0.1
• Architecture: decoder-only Llama-like Transformer
• Nominal size: 1B
• Language: Only Italian
• Canonical checkpoint: iter_1243085
• Tokenizer: Gpeik/Sophira-tokenizer-64k-v0
• License target: Apache-2.0

This release is the first completed 1B Italian base model from the Sophira project. It is intended as an open research and benchmarking artifact, and as a starting point for later downstream adaptation work.

Intended Use

This model is the first completed 1B base-pretraining artifact for the Sophira project.

Intended uses:

• Italian language modeling research
• downstream evaluation and benchmarking
• initialization for later instruction tuning or task adaptation
• reproducibility work around open Italian foundation-model pretraining

Out-of-Scope Use

This artifact is not documented or evaluated as suitable for:

• safety-critical production deployment
• legal, medical, or financial decision support
• factual-reliability-sensitive assistant use without downstream evaluation
• multilingual production use outside Italian-first evaluation

Training Data

Canonical training sources:

• uonlp/CulturaX Italian subset
• PleIAs/Italian-PD

Full-pass mixture used for this run:

• 82.977% CulturaX Italian
• 17.023% Italian-PD

Measured source-token counts:

• CulturaX_it: 135.197.223.323
• Italian_PD: 27.736.341.018
• Total: 162.933.564.341

The project license policy and source-license references remain tracked in DATA_LICENSES.md.

Training Procedure

• Training framework: Megatron-LM
• Runtime: .venv-apex
• Cluster: CINECA Leonardo. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support.
• Topology: 16 nodes / 64 GPUs
• Sequence length: 2048
• Micro-batch size: 1
• Global batch size: 64
• Tokens per step: 131072
• Target steps: 1243085
• Checkpoint interval: 8000

Final successful completion occurred on Tuesday, August 11, 2026.

Evaluation

End-of-training validation:

• iteration: 1243085
• validation loss: 2.756790E+00
• validation perplexity: 1.574921E+01

Export validation:

• Hugging Face export completed successfully
• native vs Hugging Face next-token parity passed
• parity prompt:
  • Roma, 19 luglio - Il vertice europeo sull'energia si è aperto oggi
• parity result:
  • greedy next token match restored
  • top-20 overlap restored to 20 / 20

Controlled-generation summary:

• prompts: 10
• generations: 30
• average output tokens: 65.7667
• average output words: 58.3667
• average repeated bigram fraction: 0.1017
• average repeated trigram fraction: 0.0755
• average topic keyword overlap: 0.1386
• empty generation count: 0
• malformed output count: 0
• premature stop count: 0
• eos frequency: 0.3667

Controlled-generation analysis summary:

• severity counts:
  • mixed: 23
  • good: 6
  • problematic: 1
• dominant issue types:
  • topic_drift: 20
  • repeated_clause: 6
  • high_trigram_repetition: 4
  • repeated_percentage_pattern: 1
  • repeated_sentence: 1

The main residual weakness in this release-facing prompt set is topic drift under sampled decoding, with one repetition-heavy greedy continuation on the prompt Il governo ha annunciato che.

Comparison against the released 360M baseline:

• repetition is lower in 1B than in 360M
  • bigram repetition:
    • 1B: 0.1017
    • 360M: 0.1148
  • trigram repetition:
    • 1B: 0.0755
    • 360M: 0.0809
• malformed outputs on the current prompt suite:
  • 1B: 0
  • 360M: 1
• analysis severity:
  • 1B: {'mixed': 23, 'good': 6, 'problematic': 1}
  • 360M: {'mixed': 24, 'problematic': 3, 'good': 3}
• topic drift remains the dominant residual issue in both models. Area of improvement for the next versions and adaptation.
• topical keyword overlap is lower in 1B on the current prompt suite:
  • 1B: 0.1386
  • 360M: 0.23

Release-facing takeaway:

• 1B is the stronger overall base-model release candidate on the current controlled-generation evidence
• the clearest gains over 360M are lower repetition and the absence of malformed outputs on this prompt suite
• the current evidence does not justify a strong claim that topical adherence is solved

Usage

Transformers example:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gpeik/Sophira-1B-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Il governo ha annunciato che"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation Snapshot

Completed:

• end-of-training Megatron validation
• checkpoint save/resume validation across multiple Slurm jobs
• final Hugging Face export
• native vs Hugging Face first-token parity validation
• controlled-generation evaluation
• controlled-generation analysis
• comparison against the published 360M baseline
• final checkpoint pruning to the retained release set

Areas for Improvement

• This is a base model, not an instruction-tuned model.
• Broader robustness, bias, and safety evaluation is still pending.
• This release should be treated as an open Italian base-model baseline, not as a polished assistant model.

Release Artifacts

The canonical release checkpoint is:

• iter_1243085

The canonical exported model bundle is:

• .../models/releases/Sophira-1B-base-v0.1/model

License

• Code: Apache-2.0
• Tokenizer: Apache-2.0
• Model weights target: Apache-2.0

Source dataset licenses remain governed by their upstream terms and by the repository tracking in DATA_LICENSES.md.