BLOG_PROMPT.md

# Blog Writing Prompt

Drop this entire block into a fresh Claude conversation to generate the research blog post.

---

## PROMPT

You are writing a research blog post about a reinforcement learning environment called **ToolOrchestratorEnv**. The title is **The Price of Thinking: Teaching LLM Agents When Tools Are Worth the Cost**. The authors are Andrew Lara (Franklin and Marshall College), Yashaswi Sharma (University of Southern California), Defu Cao (University of Southern California), and Muyan Weng (University of Southern California). The project builds on Yashaswi Sharma's prior SearchEconomicsEnv work. The target audience is ML researchers and practitioners who read papers like those at NeurIPS, ICLR, and the Hugging Face blog — people who understand RL and LLMs but are not experts in tool-use or search economics.

The post should be **1,500–2,000 words**, well-structured with section headers, and written in a direct, confident academic-blog tone (think: The Gradient, Hugging Face blog, or a good arXiv blog post). Avoid hype. Let the ideas do the work.

---

### What this research is and why it matters

**The core problem:** Large language model agents are increasingly given access to tools — search engines, calculators, code interpreters, databases. In real deployments, every tool call costs something: API fees, latency, rate limits, or compute. Current agent frameworks treat tools as free: call whatever you want, as many times as you want. This is unrealistic and economically wasteful.

**The research gap:** Most RL environments for tool-using agents either (a) focus on a single tool (e.g. search-only retrieval agents), or (b) ignore cost entirely and measure only answer quality. There is no standard RL training ground where the agent must *choose between tools with different price/quality tradeoffs* under a shared budget constraint.

**Actual current contribution:** This submission ships the environment, reward function, baselines, deployment artifact, and tests. It does not claim a converged trained checkpoint. The honest research status is: the benchmark is complete and ready for GRPO training; convergence and baseline-beating policy results are the next milestone.

**What we built:** ToolOrchestratorEnv — an OpenEnv-compatible RL environment that puts cost-aware tool selection at the center of the learning objective. The agent picks from six tools per step (web search, Wikipedia, calculator, Python executor, LLM reasoning, or commit) across four question domains (HotpotQA, MATH, GPQA, HumanEval), with a shared budget that depletes as tools are called.

**Why this is novel:**
- It extends the "search economics" framing from a single tool to a heterogeneous tool portfolio
- It tests transfer: can an agent learn that calculators are cheap and LLMs expensive, and route accordingly?
- The multi-domain setup forces the agent to learn *domain → tool* mappings (search for factual QA, calculator for math) rather than one-size-fits-all policies
- The Weitzman-style reward (efficiency bonus only on correct + frugal commits) creates a richer credit assignment problem than binary success/failure

---

### Background sections to write (with sources to find and cite)

**1. The tool-use agent landscape**

Explain why tool use is now central to LLM agents. Cite and discuss:
- The ReAct paper (Yao et al., 2022) — introduced interleaving reasoning and tool calls
- Toolformer (Schick et al., 2023) — self-supervised tool learning
- ToolBench / API-Bank — benchmarks for tool-using LLMs
- Find at least one recent paper (2024 or 2025) showing that tool-calling agents outperform tool-free baselines on knowledge-intensive tasks. Look at arXiv, ACL Anthology, or the Hugging Face papers page.

**2. Search economics and the budget constraint**

Explain the economic analogy: information has a cost, and rational agents should not search more than their expected marginal gain from search. Cite:
- Weitzman (1979) "Optimal Search for the Best Alternative" — the foundational search economics paper
- SearchEconomicsEnv by Yashaswi Sharma / University of Southern California (https://github.com/sharma-yash01/SearchEconomicsEnv, https://huggingface.co/spaces/yashu2000/search-economics-env) — the direct predecessor that built this RL environment for search-budget-constrained HotpotQA
- Look for any recent work on "budgeted retrieval" or "adaptive retrieval" in RAG systems (2024-2025) that shows that unconstrained retrieval hurts performance or cost-effectiveness. Papers like FLARE, IterRetGen, or similar might be relevant.

**3. The multi-domain challenge**

Explain why testing across HotpotQA, MATH, GPQA, and HumanEval matters — these domains need fundamentally different tools (search for factual, calculator for symbolic, code for algorithmic, LLM for graduate-level). Find and cite:
- The MATH benchmark paper (Hendrycks et al., 2021)
- HotpotQA paper (Yang et al., 2018)
- GPQA paper (Rein et al., 2023)
- HumanEval paper (Chen et al., 2021)
- Any paper showing that tool specialisation helps across domains (e.g., PAL, PoT, or similar)

**4. Reinforcement learning for tool selection**

Explain why RL (not just prompting or supervised learning) is the right frame for this problem: the agent must explore, face delayed rewards (only know if an answer was right after commit), and learn multi-step strategies. Cite:
- Any recent paper using RL for LLM agent training (e.g., RLHF extensions, agent-specific RL work, or OpenEnv/AgentBench)
- The OpenEnv competition framework (Berkeley RDI, AgentX) — explain what OpenEnv is and why standardised RL environments matter for reproducibility
- Look for "process reward models" or "step-level reward" papers in the agent RL space

---

### Key sections for the post

1. **The problem with free tools** — hook paragraph. Real API calls cost money. Agents don't know this. Set up the gap.

2. **Search economics, briefly** — one paragraph on Weitzman, one on SearchEconomicsEnv. The framing: information retrieval as a market with prices.

3. **ToolOrchestratorEnv: the environment** — describe the setup clearly:
   - 6 tools, 4 datasets, shared budget
   - The action-observation loop (what the agent sees, what it decides)
   - The reward formula (explain it intuitively: you pay for every call, you earn back on correct commits, and get a bonus for answering correctly without blowing your budget)
   - The Ceramic AI integration for live web retrieval

4. **Why this is hard** — explain the credit assignment problem (you don't know a tool call was wasted until you commit), the domain-routing challenge, and the exploration-exploitation tradeoff under budget pressure.

5. **Baselines and what they tell us** — describe the three baselines (random, cheapest-first, domain-oracle) and what their expected performance reveals about the structure of the problem.

6. **What we're building toward** — the research agenda: train a GRPO agent on this environment, show it beats baselines, and study what routing policies it learns. Can it learn that LLM reasoning is worth 20x the calculator cost for GPQA but wasteful for simple arithmetic?

7. **Conclusion** — the broader point: as AI systems become more agentic, cost-aware tool selection will be as important as answer quality. We need RL environments that take this seriously. This is one.

---

### Tone and style guidelines

- **Cite real papers** — do not make up citations. For any claim about related work, search arXiv, Semantic Scholar, or ACL Anthology and use the actual paper. Format citations inline as (Author et al., Year) with a references section at the end.
- **Be specific** — don't say "researchers have shown" without naming the paper.
- **Write for skeptics** — assume your reader will ask "why does this matter" and "what's actually new." Answer those questions directly in the text.
- **Avoid marketing language** — no "revolutionary," "groundbreaking," or "state-of-the-art." Just describe what was built and why it's useful.
- **Include the reward formula** — write it out mathematically and then explain it in plain English. Researchers appreciate seeing the actual math.
- **Link to the HF Space** — mention that SearchEconomicsEnv, the predecessor, is live at https://huggingface.co/spaces/yashu2000/search-economics-env and that ToolOrchestratorEnv is live at https://huggingface.co/spaces/landrew9/ToolOrchestratorEnv.

---

### What NOT to do

- Do not fabricate benchmark numbers — we don't have trained agent results yet, only baseline results. Say so honestly.
- Do not claim this is the first RL environment for tool use — be accurate about prior work.
- Do not skip the related work — proving the gap is real requires engaging with existing papers.
- Do not make the reward formula paragraph too short — this is a key technical contribution; spend time on it.

---

### Final checklist before finishing the post

- [ ] Every citation is real and can be found on arXiv or a peer-reviewed venue
- [ ] The reward formula is written out and explained in plain English
- [ ] The post explains what OpenEnv is and why deploying on HF Spaces matters
- [ ] The post mentions Ceramic AI and explains why live web retrieval matters (vs. static knowledge)
- [ ] The baseline section sets up what "winning" looks like for a trained RL agent
- [ ] A references section is included at the end with full citations