Neha12210/project1-math-reasoning-llm

🧠 Math Reasoning LLM — SFT + GRPO Fine-Tuning

Taught a small LLM to reason step-by-step using Reinforcement Learning (GRPO)

🔬 What This Project Does

Fine-tunes Qwen2.5-1.5B-Instruct (1.5B parameters) to solve math problems with chain-of-thought reasoning using a two-phase training approach:

1. Phase 1 — SFT (Supervised Fine-Tuning): Train on 50K math problems with worked solutions from NuminaMath-CoT to teach the model how to show its work
2. Phase 2 — GRPO (Group Relative Policy Optimization): Reinforce correct reasoning using reward functions — the same RL technique behind DeepSeek-R1

📊 Results

Benchmark	Base Model	After SFT	After GRPO	Improvement
MATH-500	~XX%	~XX%	~XX%	+XXpp

(Fill in after running evaluation)

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    Training Pipeline                      │
│                                                           │
│  ┌─────────────┐    ┌──────────────┐    ┌─────────────┐ │
│  │  NuminaMath  │───▶│  SFT Phase   │───▶│  SFT Model  │ │
│  │  CoT (50K)   │    │  (QLoRA)     │    │  Adapter    │ │
│  └─────────────┘    └──────────────┘    └──────┬──────┘ │
│                                                 │        │
│  ┌─────────────┐    ┌──────────────┐    ┌──────▼──────┐ │
│  │  DeepMath   │───▶│  GRPO Phase  │───▶│  GRPO Model │ │
│  │  103K       │    │  (RL + QLoRA)│    │  Adapter    │ │
│  └─────────────┘    └──────────────┘    └──────┬──────┘ │
│                                                 │        │
│  ┌─────────────┐    ┌──────────────┐    ┌──────▼──────┐ │
│  │  MATH-500   │───▶│  Evaluation  │───▶│  Benchmark  │ │
│  │  Benchmark  │    │  Script      │    │  Results    │ │
│  └─────────────┘    └──────────────┘    └─────────────┘ │
└─────────────────────────────────────────────────────────┘

🛠️ Tech Stack

Component	Tool
Base Model	Qwen/Qwen2.5-1.5B-Instruct
SFT Training	TRL SFTTrainer + PEFT QLoRA
RL Training	TRL GRPOTrainer + custom rewards
Speed Optimization	Unsloth (2x faster, 80% less VRAM)
SFT Dataset	AI-MO/NuminaMath-CoT (859K math problems)
GRPO Dataset	trl-lib/DeepMath-103K (103K prompts)
Evaluation	MATH-500 benchmark
Hardware	Kaggle T4 GPU (16GB VRAM, free tier)

📁 Project Structure

project1_math_reasoning_llm/
├── README.md                  # This file
├── 01_sft_training.py         # Phase 1: Supervised Fine-Tuning
├── 02_grpo_training.py        # Phase 2: GRPO Reinforcement Learning
├── 03_evaluation.py           # Evaluate on MATH-500 benchmark
├── kaggle_notebook_sft.py     # Ready-to-paste Kaggle notebook code
├── kaggle_notebook_grpo.py    # Ready-to-paste Kaggle notebook code
└── requirements.txt           # Dependencies

🚀 How to Run (Kaggle)

Step 1: SFT Training (~1-2 hours on T4)

1. Create a new Kaggle notebook with GPU T4 x2 accelerator
2. Copy contents of kaggle_notebook_sft.py into the notebook
3. Run all cells
4. Download the saved adapter from /kaggle/working/sft_adapter/

Step 2: GRPO Training (~2-3 hours on T4)

1. Upload the SFT adapter to a new Kaggle dataset
2. Create a new notebook with GPU T4 x2
3. Copy contents of kaggle_notebook_grpo.py
4. Update the adapter path
5. Run all cells

Step 3: Evaluation

1. Load the GRPO adapter
2. Run 03_evaluation.py to get MATH-500 scores
3. Update the results table in this README

📚 Key Concepts You'll Learn

• QLoRA: 4-bit quantized LoRA — fits large models on small GPUs
• SFT: Teaching a model to mimic expert responses
• GRPO: Group Relative Policy Optimization — RL without a separate reward model
• Reward Engineering: Designing rewards for accuracy + format + length
• Chain-of-Thought: <think>...</think> structured reasoning

📄 Based On

• DeepSeek-R1 Paper — GRPO methodology
• RL for Reasoning in Small LLMs — training recipe
• TRL Library Documentation
• Unsloth — training optimization