trl internal testing

company

Activity Feed Request to join this org

AI & ML interests

Internal testing artifact mangement for trl library

Recent Activity

kashif updated a model 5 days ago

trl-internal-testing/tiny-DiffusionGemmaForBlockDiffusion

qgallouedec new activity 6 days ago

trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration:Upload Qwen2_5_VLForConditionalGeneration

qgallouedec new activity 6 days ago

trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration:Upload Qwen2_5_VLForConditionalGeneration

View all activity

sergiopaniego

posted an update 1 day ago

Post

102

quick reminder! 🚨

tomorrow (Tuesday, July 28), we're back with Class 3 of the Training Agents live series

🧠 what: reinforcement learning for training agents (GRPO): how it works, how to implement it in TRL, and end-to-end examples
🗓️ when: Tuesday, July 28 - 🕔 5:00 PM CEST / 8:30 PM IST
📍 where: Live on @huggingface 's X, YouTube, and LinkedIn

live: https://www.youtube.com/watch?v=ztdTed5egrM

class 1: https://x.com/SergioPaniego/status/2069382207618379813
class 2: https://x.com/SergioPaniego/status/2075180665184686187

sergiopaniego

posted an update 4 days ago

Post

136

you can now train your own coding agents with trl + openenv, starting with opencode

we just added end-to-end support for training agent harnesses:

> TRL: a loop-owning training path (AsyncGRPOTrainer + HarnessRolloutWorker) that launches the agent in an OpenEnv session, reads back its trace, reconstructs the training samples, and trains with AsyncGRPO
> OpenEnv: the OpenCode harness environment plus a transparent proxy that forwards the agent's model calls and records each turn's token ids and logprobs

you train the actual opencode agent as is, it runs its own loop and tools and the policy learns from the exact tokens it produced

we're shipping a self-contained example: local subprocess sandbox, DeepCoder problems, validated on Qwen3-8B.

> example: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py
> docs: https://huggingface.co/docs/trl/main/openenv

and we're working actively on both sides so expect more 🤓

1 reply

sergiopaniego

posted an update 5 days ago

Post

1459

you can train DiffusionGemma (a block-diffusion LLM) in TRL! and we're sharing an example for it

TRL trainers are made to be easily extended and adapted to different real-world use cases.

in this one, with a single method overridden in SFTTrainer (compute_loss), you can train this model

> example: https://github.com/huggingface/trl/blob/main/examples/scripts/sft_diffusion_gemma.py

kashif

updated a model 5 days ago

trl-internal-testing/tiny-DiffusionGemmaForBlockDiffusion

Image-Text-to-Text • 4.27M • Updated 5 days ago • 13.2k • 2

qgallouedec

in trl-internal-testing/tiny-Qwen2_5_VLForConditionalGeneration 6 days ago

Upload Qwen2_5_VLForConditionalGeneration

#19 opened 6 days ago by

qgallouedec

Upload Qwen2_5_VLForConditionalGeneration

#18 opened 6 days ago by

qgallouedec

Upload Qwen2_5_VLForConditionalGeneration

#16 opened 2 months ago by

albertvillanova

Upload Qwen2_5_VLForConditionalGeneration

#17 opened 6 days ago by

qgallouedec

sergiopaniego

posted an update 6 days ago

Post

177

join us next Tuesday, July 28, for Class 3 of the Training Agents live series!

we'll dive into reinforcement learning for agent training, covering the intuition behind GRPO, how it works, and how to implement it in TRL with practical, e2e examples

see you there 🤠

live: https://www.youtube.com/live/ztdTed5egrM

> in case you missed class 1:
https://x.com/SergioPaniego/status/2069382207618379813
> and in case you missed class 2: https://x.com/SergioPaniego/status/2075180665184686187

qgallouedec

updated a model 12 days ago

trl-internal-testing/tiny-Lfm2ForCausalLM-2.5

Text Generation • 526k • Updated 12 days ago • 27.5k

qgallouedec

published a model 12 days ago

trl-internal-testing/tiny-Lfm2ForCausalLM-2.5

Text Generation • 526k • Updated 12 days ago • 27.5k

qgallouedec

updated a model 12 days ago

trl-internal-testing/tiny-Lfm2ForCausalLM

Text Generation • 526k • Updated 12 days ago • 32.6k

qgallouedec

published a model 12 days ago

trl-internal-testing/tiny-Lfm2ForCausalLM

Text Generation • 526k • Updated 12 days ago • 32.6k

sergiopaniego

posted an update 20 days ago

Post

7708

Frontier models use distillation as a step of their post-training pipelines.

In 2026 it has three jobs: compress a big model into a small one, merge RL experts into a single model, and let a model teach itself.

I wrote up which frontier models use each one and how: https://huggingface.co/blog/sergiopaniego/distillation-2026

It pairs with Class 2 of the Training an Agent series Ben and I are doing, where we teach these techniques hands-on with TRL!

3 replies

albertvillanova

posted an update 22 days ago

Post

3588

🎉 KTO is now part of the stable TRL API

As of Promote KTO to stable API, KTOTrainer and KTOConfig have graduated from trl.experimental to the stable trl API. https://github.com/huggingface/trl/pull/6175

This one closes out a long road. Over the past 6+ months, the "Align KTO with DPO" effort landed ~90 PRs methodically bringing KTO up to the standard we hold for stable trainers, one carefully-scoped change at a time:
- Feature parity with DPO: full VLM support (incl. multi-image), sync_ref_model, PEFT + Liger, ZeRO-3 + PEFT dtype fix, pad_to_multiple_of, activation offloading, IterableDataset and dict eval_dataset, remove_unused_columns, and reference-logprob precomputation at init.
- Consistency with DPO: aligned method order and signatures, tokenization, _prepare_dataset, PEFT handling, ref-model preparation for distributed training, and config layout — plus a new DataCollatorForKTO and output format. Metrics moved into _compute_loss and simplified to direct averages via the shared _metrics attribute.
- Removing legacy baggage: dropped encoder-decoder support, BOS/EOS handling, null_ref_context, generate_during_eval, model_init, preprocess_logits_for_metrics, model/ref adapter names, and several dead config knobs.
- Coverage: a full test suite mirroring DPO, text collator tests, VLM tests, and slow tests.
- The promotion itself: the experimental → stable move (#6175) and shim cleanup (#6287), handled so downstream users get a clean deprecation path.

Honestly, this has been one of the more complex tasks I've taken on since joining the team, not because any single change was hard, but because it demanded sustained consistency across a ~2,000-line trainer, with every branch, comment, and edge case kept in lockstep with DPO.

Huge thanks to everyone who reviewed along the way (especially @qgallouedec ), the incremental review cadence is exactly what kept this maintainable.

KTO now sits on equal footing with our other flagship trainers. 🚀