| | --- |
| | license: cc-by-nc-4.0 |
| | datasets: |
| | - openai/gsm8k |
| | language: |
| | - en |
| | base_model: |
| | - Qwen/Qwen2.5-Math-1.5B |
| | pipeline_tag: text-generation |
| | library_name: transformers |
| | tags: |
| | - math |
| | - qwen |
| | - lora |
| | - mathematics |
| | - gsm8k |
| | --- |
| | |
| | # OpenMath |
| | Fine-tuning a Small Language Model (SLM) for Step-by-Step Math Reasoning |
| |
|
| | ## Overview |
| | OpenMath is an open-source project focused on fine-tuning a small language model for mathematical reasoning using parameter-efficient LoRA training. |
| |
|
| | This repository contains **only a LoRA adapter** trained on the full GSM8K dataset. Users must load the base model separately and attach the adapter using PEFT. |
| |
|
| | The latest version of this model was trained on an **AMD MI300X GPU using ROCm**, demonstrating that high-performance non-NVIDIA accelerators can successfully support modern large language model fine-tuning with PyTorch and Hugging Face. |
| |
|
| | --- |
| |
|
| | ## Base Model |
| | **Qwen/Qwen2.5-Math-1.5B** |
| |
|
| | This repository **does not contain the base model weights** — they must be loaded directly from Hugging Face before applying this LoRA adapter. |
| |
|
| | --- |
| |
|
| | ## Hardware Used (Latest Training Run) |
| |
|
| | - **GPU:** AMD MI300X (ROCm 7.0) |
| | - **VRAM:** 192 GB |
| | - **OS:** Ubuntu 24.04 |
| | - **Framework:** PyTorch + Hugging Face |
| | - **Backend:** ROCm |
| |
|
| | --- |
| |
|
| | ## Dataset |
| |
|
| | **GSM8K (Grade School Math 8K)** |
| | - **Training samples:** 7,473 (full training split) |
| | - **Evaluation:** Full GSM8K test split (1,319 problems) |
| |
|
| | Only the solution portion of each example was used for loss computation via loss masking to encourage stronger reasoning behavior. |
| |
|
| | --- |
| |
|
| | ## Training Configuration |
| |
|
| | **Method:** LoRA (full precision, bfloat16) |
| | **Precision:** bfloat16 (no 4-bit quantization in this run) |
| |
|
| | ### LoRA settings |
| | - Rank: 16 |
| | - Alpha: 32 |
| | - Dropout: 0.05 |
| | - Target modules: `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| |
|
| | ### Data & sequence |
| | - Max sequence length: 1024 |
| |
|
| | ### Optimization |
| | - Per-device batch size: 2 |
| | - Gradient accumulation: 8 |
| | - Effective batch size: 16 |
| | - Learning rate: 1e-4 |
| | - Optimizer: `adamw_torch` |
| | - Scheduler: cosine |
| | - Warmup: 5% |
| |
|
| | ### Training |
| | - **Epochs:** 3 |
| |
|
| | --- |
| |
|
| | ## Results |
| |
|
| | **GSM8K Accuracy (Full Test Set):** |
| | 750 / 1319 = **56.86% accuracy** |
| |
|
| | This represents a substantial improvement over earlier small-scale Colab experiments and is a strong result for a 1.5B model trained with LoRA on the full dataset. |
| |
|
| | --- |
| |
|
| | ## GSM8K Accuracy Comparison |
| |
|
| | | Model | Accuracy (%) | |
| | |---------------------------|-------------| |
| | | Qwen1.5-7B | 62.50 | |
| | | Param2-17B-A2.4B-Thinking | 57.32 | |
| | | **OpenMath** | **56.86** | |
| | | Llama2-70B | 56.80 | |
| | | Llama-3-8B | 56.00 | |
| | | Mistral-7B | 52.20 | |
| | | Gemma-7B | 46.40 | |
| | | DeepSeek-V2-Lite | 38.21 | |
| | | gpt-oss-20b | 36.54 | |
| |
|
| |
|
| |  |
| |
|
| | --- |
| |
|
| | ## How to Use This Model |
| |
|
| | 1. Load the base model **Qwen/Qwen2.5-Math-1.5B** from Hugging Face. |
| | 2. Attach this LoRA adapter using PEFT. |
| | 3. Use a structured prompt that includes an instruction, problem, and solution section for best results. |
| |
|
| | --- |
| |
|
| | ## Why This Matters |
| |
|
| | - Demonstrates that **AMD MI300X** can effectively train modern LLMs with Hugging Face + LoRA. |
| | - Shows strong math reasoning at **1.5B parameters** with lightweight fine-tuning. |
| | - Provides a compact adapter instead of requiring users to download a massive full model. |
| |
|
| | --- |
| |
|
| | ## Limitations |
| |
|
| | - The model can make reasoning mistakes. |
| | - It should not be used for exams, assignments, or professional decisions. |
| | - Performance depends heavily on prompt formatting. |
| |
|
| | --- |
| |
|
| | ## License |
| |
|
| | **cc-by-nc-4.0** |
| |
|
