tksluangrath/tool-concierge

Tool Concierge (Qwen3-8B-Base + LoRA for MCP Tool-Call Generation)

1. Introduction

The Model Context Protocol (MCP) became a cross-vendor standard "in a very short time span" (Anthropic, 2024). Models that hallucinate tool names or malform arguments turn these integrations into silent failures (Patil et al., 2023). Routing every tool-use decision through a frontier model is also costly at scale, and small open-weight models fine-tuned on curated function-calling data can match or beat larger ones (Erdogan et al., 2024). I fine-tuned Qwen/Qwen3-8B-Base with LoRA to generate a single MCP tool call, or an explicit no_applicable_tool abstention, from a plain-English request and a set of MCP tool definitions. Gorilla (Patil et al., 2023) trains against ML model-hub APIs, and TinyAgent (Erdogan et al., 2024) targets generic JSON tool calls; I narrowed scope instead to the MCP schema and treated abstention as a trained output rather than a fallback. Fine-tuning raised own-test-split call accuracy from 523/929 to 698/929 and abstention accuracy from 9/31 to 22/31.

2. Data

I pulled positive training rows from Salesforce/xlam-function-calling-60k (60,000 verified examples across 3,673 APIs and 21 domains), sampling 9,300 of them (random_state=5002) and reformatting xlam's inline tool schema into MCP-style inputSchema JSON, with each answers[0] reshaped into a single {"name":..., "arguments":...} call. I self-generated the negative (no-applicable-tool) examples, but only landed 300 cleaned records in negatives_data.json instead of the roughly 4,000 I'd planned - the generator model had to be swapped from a 70B to a 14B model after a disk-quota issue, which cut the negative target "to fit a hard deadline." That left about a 97/3 positive/negative split instead of the planned 70/30. I shuffled the combined set (frac=1, random_state=5002) and split it 80% train / 10% eval / 10% test, producing train_split.json, eval_split.json, and a 960-record test_split.json; the actual training run further subsampled to 2,500 train rows and 300 eval rows (random_state=5002) for time.

3. Methodology

I chose LoRA for compute efficiency on a single A100, applying adapters to q_proj and v_proj (bias="none", task_type="CAUSAL_LM") across three hyperparameter combinations, each trained for 1 epoch at learning_rate=1e-4 with batch size 1 (train and eval), 2048-token max sequence length, and evaluation/checkpointing every 1000 steps with load_best_model_at_end=True. The three combos I tested were r=64, alpha=64, dropout=0.05 (combo1_high_capacity), r=16, alpha=16, dropout=0.10 (combo2_low_capacity), and r=32, alpha=64, dropout=0.15 (combo3_mid_capacity). combo1_high_capacity won on final eval loss (0.2866 vs. 0.2868 for combo3 and 0.3099 for combo2), so I saved its adapter and used it for all reported evaluation results.

4. Evaluation

I used four benchmarks to cover different failure modes. BFCL scored tool-call correctness and abstention on a mix of simple and irrelevance examples, Nexus scored single-turn function calling against real-world API definitions, MMLU served as a catastrophic-forgetting check with a plain multiple-choice prompt and no scaffold, and own_test_split scored the model on my own held-out MCP-formatted test rows. I don't have comparison-model benchmarks in this same suite - I ran a smaller in-context comparison of Qwen2.5-1.5B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-14B-Instruct in CheckIn3, but on only 2 fixed examples with call-accuracy scoring, so it doesn't map onto this table. Fine-tuning produced a large gain on own_test_split abstention accuracy, a sizable gain on Nexus, a smaller gain on BFCL, and a small drop on MMLU.

Benchmark	This Model (posttrain)	Base Model (pretrain)	Comparison Model 1	Comparison Model 2
BFCL	call_acc 67/100, abstention_acc 72/100, false_call 28/100	call_acc 58/100, abstention_acc 44/100, false_call 22/100	TODO -- no comparison model benchmark data found in source files	TODO -- no comparison model benchmark data found in source files
Nexus	call_acc 142/200	call_acc 43/200	TODO -- no comparison model benchmark data found in source files	TODO -- no comparison model benchmark data found in source files
MMLU	acc 107/500 (21.4%)	acc 121/500 (24.2%)	TODO -- no comparison model benchmark data found in source files	TODO -- no comparison model benchmark data found in source files
own_test_split	call_acc 698/929, abstention_acc 22/31	call_acc 523/929, abstention_acc 9/31	TODO -- no comparison model benchmark data found in source files	TODO -- no comparison model benchmark data found in source files

5. Usage and Intended Uses

This model handles single-step MCP tool-call generation only; it isn't built for multi-step orchestration or planning loops. Load it as a PEFT/LoRA adapter on top of the Qwen3-8B-Base checkpoint, using the snippet below.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen3-8B-Base"
adapter_repo_id = "tool-concierge"  # replace with your-username/tool-concierge

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, adapter_repo_id)

6. Prompt Format

I used the fixed "Least-to-Most Prompting" scaffold below (Zhou et al., 2022) for every training row and eval prompt, except MMLU, which used a plain multiple-choice prompt instead.

SCAFFOLD_TEMPLATE = """\
You generate a single MCP tool call for a user request. Work through the steps in order, using the answer to each step in the next one.
Step 1. Restate the user's goal in one sentence.
Step 2. List the available tools. For each tool, check its inputSchema and decide whether it can satisfy the goal.
Step 3. Pick the one tool that fits, or decide that none apply.
Step 4. If a tool fits, read its inputSchema and pull each argument value from the request, matching the schema's types and including every required field. If no tool fits, prepare an abstention.
Step 5. Output the final answer as a single JSON object and nothing else. For a tool call: {{"name": <tool>, "arguments": {{...}}}}. If no tool applies: {{"name": "no_applicable_tool", "arguments": {{"reason": <short reason>}}}}.

User request:
{query}

Available MCP tools:
{tools_json}"""

7. Expected Output Format

The model's final line is a single JSON object with a tool name and typed arguments. Below is a real posttrain example from posttrain_samples.json (own_test_split[0]), shown as-is, including its recorded correct: false score:

{"name": "generate_random_string", "arguments": {"length": 10, "uppercase": true, "lowercase": true, "digits": true}}

8. Limitations

MMLU accuracy dropped from 121/500 (24.2%) to 107/500 (21.4%) after fine-tuning. Both numbers sit close to the 25% chance rate for four-option questions, so the signal is weak, though the drop is directionally consistent with LoRA's catastrophic-forgetting risk. BFCL's false-call rate rose from 22/100 to 28/100 even as abstention accuracy improved to 72/100: pretrain only resolved 66/100 irrelevance examples to a parseable abstain-or-false-call output at all, versus all 100 posttrain, so I read this as an output-format fix rather than better abstention judgment. own_test_split abstention accuracy improved from 9/31 to 22/31, but 9 of 31 cases were still wrong. My training corpus ended up at roughly 9,300 positive to 300 negative rows instead of the ~70/30 mix I'd planned, the split ended at 80/10/10 instead of the planned 80/20, and I subsampled the final run to 2,500 train rows and 300 eval rows - all deadline-driven. I only benchmarked the winning combo (combo1) post-training, so I couldn't compare the MMLU drop across combos. The Nexus call-accuracy jump (43/200 to 142/200) was much larger than BFCL's (58/100 to 67/100); this could be specific to Nexus's toolset format or parser rather than a uniform training effect. A smaller CheckIn3 pilot BFCL run (n=40) produced different pretrain numbers (19/40 call_acc, 22/40 abstention_acc) than the n=100 run reported here - I'm chalking that up to sample-size variance. These results only test the MCP schema framing. I didn't run a plain-function-calling ablation to compare against it, which matters given prior work questioning whether MCP reliably outperforms plain function-calling (Luo et al., 2025).

Citations

Anthropic. (2024, November 25). Introducing the Model Context Protocol. https://www.anthropic.com/news/model-context-protocol

Erdogan, L. E., Lee, N., Jha, S., Kim, S., Tabrizi, R., Moon, S., Hooper, C., Anumanchipalli, G., Keutzer, K., & Gholami, A. (2024). TinyAgent: Function calling at the edge. arXiv. https://arxiv.org/abs/2409.00608

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv. https://arxiv.org/abs/2106.09685

Luo, Z., Shi, X., Lin, X., & Gao, J. (2025). Evaluation report on MCP servers. arXiv. https://arxiv.org/abs/2504.11094

Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large language model connected with massive APIs. arXiv. https://arxiv.org/abs/2305.15334

Qwen Team. (2024). Qwen3-8B-Base [Model]. Hugging Face. https://huggingface.co/Qwen/Qwen3-8B-Base

Salesforce. (n.d.). xlam-function-calling-60k [Dataset]. Hugging Face. https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k

Zhou, D., et al. (2022). Least-to-most prompting enables complex reasoning in large language models. arXiv. https://arxiv.org/abs/2205.10625