TanitAI/Tanit-MedReason-8B

Tanit-Med-8B

An 8B medical reasoning model that thinks before it answers — and a model card that tells you where it doesn't.

Tanit-Med-8B is a full fine-tune of Qwen3-8B for clinical multiple-choice reasoning and medical question answering. It was trained in four stages — broad medical SFT, reasoning SFT, DPO, and a short chain-of-thought polish — and evaluated across the ten benchmarks in MedAgentsBench, on both the standard and the hard splits.

It is named for Tanit, the Carthaginian goddess who watched over the western Mediterranean. The name is a commitment as much as a nod: this model is built by a North African team, and AfriMedQA is a first-class benchmark here, not a footnote.

The short version: on the standard splits, Tanit-Med-8B is, to our knowledge, the strongest open 8B medical model on MedAgentsBench — a 50.7% macro average, against 34.6% for the next-best 8B peer. On the hard splits it is not meaningfully better than any other 8B model, and neither is anyone else. We think both halves of that sentence matter, so both are in this card.

---

At a glance

Base model	Qwen/Qwen3-8B (dense, 8.2B params)
Precision	BF16
Context	32K native (trained at 8,192-token packed sequences)
Reasoning	<think> blocks, always on by default (≈99% of responses)
Language	English (only language evaluated)
License	Apache-2.0
Best at	Multiple-choice clinical QA, USMLE-style vignettes, medical exam reasoning
Not for	Diagnosis, treatment, dosing, triage, or anything touching a real patient

---

Results

All numbers below are zero-shot, greedy decoding, using the MedAgentsBench prompt and answer-index extraction. FULL is the standard test split; HARD is the MedAgentsBench hard subset. Baseline numbers for the other 8B models were produced by us under the same harness.

Against open 8B medical models

Benchmark	Tanit-Med-8B	DeepSeek-R1-0528-Qwen3-8B	Falcon-H1R-7B	HuatuoGPT-o1-8B	MedReason-8B	Ministral-3-8B-Reasoning
MedQA	65.0	44.0	28.9	29.5	28.0	27.8
PubMedQA	68.0	60.2	60.0	55.2	55.2	57.2
MedMCQA	57.8	42.9	33.6	35.8	35.0	34.3
MedBullets	42.9	26.0	20.1	20.5	19.2	17.2
MMLU (med)	77.5	50.5	30.3	26.6	22.8	22.2
MMLU-Pro (med)	47.3	20.7	13.6	13.6	14.4	9.4
MedExQA	73.3	49.3	35.1	28.7	21.7	21.7
MedXpertQA-R	12.1	10.5	9.6	11.4	18.1	11.0
MedXpertQA-U	14.6	9.7	10.4	11.5	16.5	9.7
AfriMedQA	48.9	32.2	25.3	12.6	12.6	10.9
Macro avg	50.7	34.6	26.7	24.5	24.4	22.1

Tanit-Med-8B wins eight of ten. The two it loses are the two where nobody is really winning (see below).

The AfriMedQA gap is the one we're proudest of: 48.9 vs 32.2 for the strongest baseline, and 4× the score of HuatuoGPT-o1 and MedReason. Medical models trained on US-exam corpora tend to fall over on questions grounded in African clinical practice. This one falls over less.

Against frontier models, for scale

These are the published MedAgentsBench reference numbers (arXiv:2503.07459), not re-run by us. Cross-harness comparison, so read them as a ruler rather than a leaderboard. Averaged over the nine benchmarks the paper reports.

Model	FULL (9-bench avg)	HARD (9-bench avg)
DeepSeek-R1	73.9	32.5
o3-mini	71.8	28.0
GPT-4o	68.2	18.0
o1-mini	67.8	25.3
DeepSeek-V3	63.1	12.2
Claude-3.5-Sonnet	61.8	12.3
Llama-3.3-70B	61.8	12.4
QwQ-32B	61.0	17.1
GPT-4o-mini	57.8	10.8
Claude-3.5-Haiku	55.0	12.0
Tanit-Med-8B	50.9	20.4

An 8B model running on a single consumer GPU lands about 5 points behind GPT-4o-mini and 4 behind Claude-3.5-Haiku on the standard splits. That is the honest position: competitive with last-generation small frontier models, not with frontier models.

Please do not read the HARD column as "an 8B model beats GPT-4o." See the next section for why that number is a mirage.

---

How to read the hard splits (please read this)

The MedAgentsBench hard subsets were built by keeping only questions that fewer than half of a panel of baseline LLMs answered correctly — a panel that included GPT-4o. Those models are therefore adversarially selected against on this split. GPT-4o's 18.0 is not a measurement of GPT-4o being worse at medicine than an 8B model; it is a measurement of the filter having done its job.

Three things follow, and we'd rather say them ourselves than have someone say them in the community tab:

1. At 8B, the hard splits do not separate models. Here is every 8B-class model we tested, macro-averaged over the ten hard subsets:

Model	HARD macro avg
Falcon-H1R-7B	21.5
MedReason-8B	21.2
HuatuoGPT-o1-8B	20.7
Tanit-Med-8B	20.3
Ministral-3-8B-Reasoning	19.6
DeepSeek-R1-0528-Qwen3-8B	17.4

A 4-point spread across six models with wildly different training. Our +16-point advantage on the standard splits evaporates entirely. We do not claim a hard-split win, because there isn't one.

2. The samples are tiny. AfriMedQA-hard is 32 questions — one item is worth 3.1 points. Seven of the ten hard subsets are n=100. Treat any difference under ~8 points on these splits as noise, ours included.

3. Bigger doesn't fix it, and self-consistency can make it worse. In our own baseline sweep, Qwen3-30B-A3B with sampling + self-consistency (k=3) reached 60.2% micro-average on the standard splits and 14.9% on the hard ones — below Qwen3-0.6B's 20.2%. On adversarially-filtered items, majority voting appears to converge confidently on the attractive distractor. This was the most useful negative result of the project and it is why we ship a greedy, single-sample model.

---

Where this model fails

• MedXpertQA is unsolved at this scale. Its items carry up to ten answer options, which puts chance around 10%. We score 12.1 (R) and 14.6 (U) — barely off the floor. So does every other 8B model here. DeepSeek-R1 gets 37.3. This is not a benchmark we are competitive on; it's a benchmark that shows where 8B runs out of road.
• MedReason-8B beats us on both MedXpertQA subsets (18.1 / 16.5). Knowledge-graph-grounded training seems to buy something on deep-reasoning items that our curriculum doesn't.
• Hard splits: no better than the field. See above.
• Long-form clinical advice is untested. Every number in this card comes from multiple-choice benchmarks. We have not evaluated free-text safety, hedging, refusal behaviour, hallucinated citations, or drug dosing.
• English only. AfriMedQA is English-language. We have run no non-English evaluation.
• Answer-format brittleness. The phase-4 checkpoint is tuned to emit a strict final-answer line. Prompt it off-format and extraction gets flaky (see Evaluation protocol).

A note we owe you

We have not yet published a like-for-like Qwen3-8B baseline under this exact frozen harness. Our internal sweep of Qwen3-8B in thinking mode lands in a similar range to Tanit-Med-8B on several standard splits, and we are not going to make a claim of the form "medical fine-tuning beats the base model" until that number is measured under the same conditions as everything else in this card. Until it is up, read the peer table as a comparison against other medical fine-tunes, not as proof that medical SFT was worth it. That number is coming, and it will go here whatever it says.

---

Training

Four stages, full fine-tune (no LoRA), ZeRO-3 / FSDP, bf16.

Phase	Method	Data	Purpose
1	SFT	medical-o1-reasoning (Huatuo-o1) + Medical-R1-Distill + MedReason	Broad medical foundation, clinical language
2	SFT	m23k + II-Medical-RL + ChatDoctor-RL + MedReason	MCQ robustness, multi-step reasoning structure
3	DPO	FineMed-DPO (32.9K pairs)	Preference alignment; fewer confidently-wrong answers
4	SFT	MedReason + Huatuo-o1, outputs rewritten to ≤6 reasoning bullets + final answer line	Concise, extractable chain-of-thought

Hyperparameters

Optimizer	AdamW, β = (0.9, 0.95), ε = 1e-8, weight decay 0.1
Grad clip	1.0
Max sequence length	8,192, packing on
Attention	FlashAttention
Global batch	~1,048,576 tokens/step (≈128 × 8,192 seqs), grad-accum to match
Schedule	2% warmup, cosine decay to 10% of peak
Peak LR — phase 1	2.0e-5
Peak LR — phase 2	1.0e-5
Peak LR — phase 3 (DPO)	5.0e-7
Peak LR — phase 4	5.0e-6

What each phase actually bought

Measured on our internal eval harness, macro-averaged across all ten standard splits:

• Phase 1 → 2 was the single biggest jump. Reasoning SFT took MedBullets from 30.5 to 47.1 and MMLU-Pro from 25.7 to 43.8 — the compositional benchmarks, exactly where you'd hope forced multi-step reasoning would pay.
• Phase 3 (DPO) cost raw accuracy. The DPO checkpoint sits several points below phase 4 on the standard splits and only reasons on 40–80% of prompts. What it bought was calibration: markedly fewer confident wrong answers. We ship it separately as Tanit-Med-8B-DPO because for some downstream uses that trade is the right one.
• Phase 4 (CoT polish) recovered the accuracy and pushed the reasoning rate to ~99% of responses while shortening the traces. It also cut extraction failures by an order of magnitude, which — see below — turned out to matter more than we expected.

We capped the reasoning budget at 4,096 tokens throughout. This follows m1, which identifies an optimal medical-reasoning threshold around 4K tokens, past which accuracy declines — unlike in maths, forcing more reasoning on a medical question mostly gives a model with a shaky knowledge prior more rope to talk itself out of a correct answer. We saw the same thing and stopped fighting it.

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TanitAI/Tanit-Med-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [
    {"role": "system", "content": "You are Tanit, an expert medical AI assistant."},
    {"role": "user", "content": (
        "A 45-year-old presents with acute monoarthritis of the first MTP joint.\n"
        "What is the most likely diagnosis?\n\n"
        "A. Gout\nB. Pseudogout\nC. Septic arthritis\nD. Rheumatoid arthritis"
    )},
]

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))

Serving with vLLM:

vllm serve TanitAI/Tanit-Med-8B --max-model-len 32768 --reasoning-parser qwen3

Generation settings. For open-ended use, follow Qwen3's thinking-mode defaults: temperature=0.6, top_p=0.95, top_k=20. For benchmark reproduction, use greedy (temperature=0) — every number in this card was produced that way. Do not use greedy decoding with enable_thinking=True for long open-ended generations; it degenerates into repetition, as it does for the base model.

Quantized: Tanit-Med-8B-NVFP4 runs on a single consumer GPU.

---

Evaluation protocol

Reproducing our numbers requires matching three things, in descending order of how much they'll bite you:

1. Answer extraction. This is a bigger confound than most training decisions. Between two revisions of our own harness, the strict-format extraction-failure rate on this checkpoint moved from 31.3% → 4.8% on MedQA (and 63% → 5% on MedQA-hard), while accuracy moved by about a point. If you benchmark against us and get a different number, check your extraction rate first. Ours is under 7% on every split.
2. Prompt. We use the MedAgentsBench zero-shot prompt verbatim — knowledgeable-medical-assistant system message, question, options, "reply with the answer index only."
3. Decoding. Greedy. Single sample. Reasoning budget 4,096 tokens, generation cap 16,384.

Self-consistency (k=5) is worth roughly +3–10% relative on the standard splits if you can afford 5× the compute — but re-read the hard-split warning above before you assume it helps everywhere. It doesn't.

---

The collection

Model	Stage	Use it if
Tanit-Med-8B	Phase 4 (CoT polish)	You want the best accuracy. Start here.
Tanit-Med-8B-DPO	Phase 3 (DPO)	You'd trade a few points of accuracy for better-calibrated confidence
Tanit-MedReason-8B	Phase 2 (reasoning SFT)	You want an un-DPO'd reasoning checkpoint to build on
Tanit-Med-8B-NVFP4	Phase 4, FP4	You're deploying on a consumer GPU

---

Intended use and limitations

Intended for: medical NLP research, benchmark development, medical education tooling, retrieval-augmented clinical QA prototypes, and as a base for further fine-tuning.

Not intended for, and not safe for: clinical decision support, diagnosis, treatment or dosing recommendations, triage, or any patient-facing deployment. Tanit-Med-8B is not a medical device. It has not been reviewed or cleared by any regulator, has not been evaluated for clinical safety, and has been measured only on multiple-choice exam questions — which correlate with medical knowledge but are not a proxy for clinical judgment. It answers with a <think> trace that looks like reasoning; a confident, fluent, well-structured trace can and does precede a wrong answer.

Known risks: inherits the biases of its training corpora, which are overwhelmingly Western and exam-derived. Will hallucinate drug names, doses, and citations. Will not reliably refuse out-of-scope or unsafe requests — DPO here optimized for correctness under ambiguity, not for safety refusals.

If you deploy anything downstream of this model in a setting where a person could be harmed, that safety work is yours to do, and it has not been done here.

---

Licensing

Model weights are released under Apache-2.0, inherited from Qwen3-8B. Note that the training corpora carry their own terms — several are research-oriented and some derive from sources with non-commercial restrictions. If you intend to use this model commercially, verify the licence of each dataset listed in the metadata; we are releasing weights, not indemnity.

---

Citation

@misc{tanit-med-8b-2026,
  title  = {Tanit-Med-8B: A Four-Stage Curriculum for Open Medical Reasoning at 8B},
  author = {Tanit Healthcare Technologies},
  year   = {2026},
  url    = {https://huggingface.co/TanitAI/Tanit-Med-8B}
}

Built on Qwen3 (Qwen Team). Evaluated with MedAgentsBench (Zhang et al., 2025). Trained on data from FreedomIntelligence, UCSC-VLAA, Intelligent-Internet, and FineMed. Our thanks to all of them — the open medical-AI stack is a shared one.

Found a problem with these numbers? Open a discussion. We'd rather be corrected than cited wrongly.