DRM-T5-Base-quartz

This model is a fine-tuned version of google-t5/t5-base trained on the QuaRTz dataset.

This model is a part of the artifact release for the research paper: Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking.

Paper: https://arxiv.org/abs/2505.23117
Repository: https://github.com/yophis/decom-renorm-merge

Uses

The model can be loaded as follows:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "yophis/DRM-T5-Base-quartz"

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

# Load the model
model = AutoModelForSequenceClassification.from_pretrained(model_id, device_map="auto")
model.config.pad_token_id = model.config.eos_token_id

# Input template
input_text = "Question: {question} Choices: (A) {choice0} (B) {choice1} Context: {context}"

Training Details

Training Data

We finetune the model on QuaRTz dataset.

Training Hyperparameters

• Learning Rate: 1e-4
• Weight Decay: 0.0
• Training Steps: 50000
• Batch Size: 1024
• Precision: bf16 mixed precision

Citation

If you find this model useful, please consider citing our paper:

@article{chaichana2025decom,
  title={Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking},
  author={Chaichana, Yuatyong and Trachu, Thanapat and Limkonchotiwat, Peerat and Preechakul, Konpat and Khandhawit, Tirasan and Chuangsuwanich, Ekapol},
  journal={arXiv preprint arXiv:2505.23117},
  year={2025}
}

Please also cite QuaRTz and the original T5 model.