ScarletMT-Nano

A Japanese → English translation model (BART-based seq2seq), trained from scratch using the Hugging Face Trainer.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Yokii2/ScarletMT-Nano")
model = AutoModelForSeq2SeqLM.from_pretrained("Yokii2/ScarletMT-Nano")

inputs = tokenizer("ノバスコシア州ハリファックスにあるダルハウジー大学医学部教授でカナダ糖尿病協会の臨床・科学部門の責任者を務めるエフード・ウル博士は、この研究はまだ初期段階にあるとして注意を促しました。", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

The model was evaluated on the Yokii2/jmdict-ja-en-bench benchmark (divided into short, medium, and long splits) using sacrebleu (BLEU, chrF2) and COMET22.

The table below displays the performance comparison with other models, ordered from highest to lowest overall performance:

Model	Model Path	Benchmark	BLEU	chrF2	COMET22
Custom (v2)	Yokii2/quickmt-ja-en-v2	short	40.68	61.71	87.50
Custom (v2)	Yokii2/quickmt-ja-en-v2	medium	39.95	65.36	89.86
Custom (v2)	Yokii2/quickmt-ja-en-v2	long	37.52	64.64	88.38
Base	quickmt/quickmt-ja-en	short	33.45	53.83	84.35
Base	quickmt/quickmt-ja-en	medium	33.88	59.42	87.81
Base	quickmt/quickmt-ja-en	long	31.82	59.87	86.75
ScarletMT-Nano	Yokii2/ScarletMT-Nano	short	31.20	53.99	82.07
ScarletMT-Nano	Yokii2/ScarletMT-Nano	medium	30.15	58.87	86.50
ScarletMT-Nano	Yokii2/ScarletMT-Nano	long	29.03	58.80	85.33
OPUS-MT	Helsinki-NLP/opus-mt-ja-en	short	25.91	46.92	80.42
OPUS-MT	Helsinki-NLP/opus-mt-ja-en	medium	22.15	49.13	82.84
OPUS-MT	Helsinki-NLP/opus-mt-ja-en	long	20.26	48.65	81.15