Model Card for Qwen2-0.5B-Instruct-SQL-generator

This model is a fine-tuned version of Qwen/Qwen2-0.5B-Instruct.
It has been trained using TRL (Transformer Reinforcement Learning) for SQL generation tasks.

Quick Start

from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="onkolahmet/Qwen2-0.5B-Instruct-SQL-generator", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])

Training Procedure

This model was trained with Supervised Fine-Tuning (SFT) using the gretelai/synthetic_text_to_sql dataset.
The goal was to fine-tune the model to better translate natural language queries into SQL statements.

Framework Versions

• TRL: 0.12.2
• Transformers: 4.46.3
• PyTorch: 2.6.0
• Datasets: 3.4.1
• Tokenizers: 0.20.3

Evaluation Results

The model was evaluated using standard text generation metrics (BLEU, ROUGE-L F1, CHRF) in both zero-shot and few-shot prompting scenarios.

🔹 Zero-shot Prompting (on gretelai/synthetic_text_to_sql/test)

After Post-processing:

• BLEU Score: 0.5195
• ROUGE-L F1: 0.7031
• CHRF Score: 70.0409

Before Post-processing:

• BLEU Score: 0.1452
• ROUGE-L F1: 0.3009
• CHRF Score: 47.8182

SQL-Specific Metrics:

• Exact Match (case insensitive): 0.1600
• Normalized Exact Match: 0.1500
• Average Component Match: 0.4528
• Average Entity Match: 0.8807

Query Quality Distribution:

• High Quality (≥80% component match): 18 (18.0%)
• Medium Quality (50-79% component match): 28 (28.0%)
• Low Quality (<50% component match): 54 (54.0%)

---

🔹 Few-shot Prompting (on gretelai/synthetic_text_to_sql/test)

After Post-processing:

• BLEU Score: 0.2680
• ROUGE-L F1: 0.4975
• CHRF Score: 57.1704

Before Post-processing:

• BLEU Score: 0.1272
• ROUGE-L F1: 0.2816
• CHRF Score: 46.1643

SQL-Specific Metrics:

• Exact Match (case insensitive): 0.0000
• Normalized Exact Match: 0.0000
• Average Component Match: 0.2140
• Average Entity Match: 0.8067

Query Quality Distribution:

• High Quality (≥80% component match): 4 (4.0%)
• Medium Quality (50-79% component match): 17 (17.0%)
• Low Quality (<50% component match): 79 (79.0%)

Citations

Cite TRL as:

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}