Embarrassingly Simple Self-Distillation Improves Code Generation
Abstract
Simple self-distillation improves code generation in large language models by fine-tuning on model-generated samples, effectively addressing precision-exploration trade-offs in decoding.
Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation.
Community
This is awesome!
I wonder if some filtering could improve training metrics even beyond what you've found, such as using a cheap proxy model to disambiguate outputs as "helpful" and "not so helpful".
Correctness is not the same thing as usefulness. That is clear. So using an equivalent fewer-parameter model would be especially relevant. (Not a classifier, but a training test-run that gives useful signals i.e., loss & evals.)
Quick follow-up after actually trying this direction:
I tested a variant where a small proxy model (sub-1B) is used to score generations by "usefulness" (rather than correctness), and then used that to filter SSD data.
On my setup (~3B student, ~0.5B proxy), proxy-filtered SSD consistently beat both raw SSD and correctness filtering on HumanEval+ (by ~13-21 points depending on baseline). So at least in this regime, it does seem like there's real signal in 'usefulness ≠ correctness', and a cheap model can pick it up!
One interesting detail: a lot of the high-utility samples selected by the proxy were actually incorrect/non-executable, but still seemed to improve downstream training. So, just like your guy's paper found, it's not about finding hidden correct solutions, but structural utility.
Super cool, in my opinion.
Caveat: none of the fine-tuned variants beat the base model yet, so this feels more like a better SSD selection rule than a full win over the base setup... Also still working through LiveCodeBench, which should be a better test of whether this holds up on harder problems.
But I'm sure if an actual lab looked into this they'd probably be able to make it work with much more general improvement. I'm just a single guy.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- On-Policy Self-Distillation for Reasoning Compression (2026)
- HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation (2026)
- Entropy-Aware On-Policy Distillation of Language Models (2026)
- Shorter Thoughts, Same Answers: Difficulty-Scaled Segment-Wise RL for CoT Compression (2026)
- Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation (2026)
- ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning (2026)
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Interesting premise. Sounds similar to some of the refining techniques that were coming out last year, maybe the mechanism is different? Only had time to skim so far. Sounds promising based on current results.
Get this paper in your agent:
hf papers read 2604.01193 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper