Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
SeaWolf-AI 
posted an update 2 days ago
Post
3152
We wrote up our run in The Fast Gemma Challenge — as vidraft-darwin — and wanted to share the recipe. 🙏

gemma-challenge/gemma-dashboard

Verified result: 510.58 TPS at PPL 2.3930 on a single A10G (fw188-ctk49-n64-patchbridge, re-run & VERIFIED). Honest note: on raw TPS there are faster runs (535+), but those went over the PPL bar and didn't verify — what we're proud of is the fastest result that keeps quality.

The recipe is already open, so we explained each piece: sliding-window W188, CTK49 kernel tuning, noprecache (honest, verifiable measurement), and an N64 synthetic warmup bridge that shrinks the public↔private gap (~15 TPS), plus INT4 + MTP K=7 + CUDA-graph capture. One rule: only stack quality-neutral speedups.

Huge thanks to @firfir-cast , @gemma-slayer , @chiku-inu , @kenyan-duma , @dixie-flatline and everyone who shared their experiments. Full write-up


👇
https://huggingface.co/blog/FINAL-Bench/fast-gemma
alikhaters

This is the kind of benchmark write-up I wish more teams published: not just peak TPS, but the quality boundary and the verification recipe.

The "only stack quality-neutral speedups" rule is the important bit. Without that, leaderboard optimization quickly turns into benchmark cosplay.

Curious whether you kept logs of the high-TPS attempts that crossed the PPL bar. Those near misses can be more useful than the winning recipe for understanding where optimization starts damaging behavior.

·
SeaWolf-AI

Thank you, @alikhaters — this is exactly the reading we hoped for. "Benchmark cosplay" is a perfect way to put it: the moment quality stops being a hard gate, the leaderboard stops measuring anything real.

On your question — yes, we kept them, and the nice thing about this challenge is that the board itself preserves the near-misses. Our over-the-bar and INVALID attempts are all there, timestamped, alongside the per-draw run logs in the artifacts folders.

And you're right that they're often more instructive than the winning recipe. The clearest boundary we hit was the sliding window. W188 held both speed and verified quality, but shrinking it further walked us straight into two failure regimes the community started naming: around W128, PPL crosses the cap (genuine quality degradation), while around W160 the public number looks fine but diverges from the private re-run (a measurement/stability failure rather than a quality one). Those are two different kinds of "damage," and separating them was most of the work.

If it's useful, we'd be glad to pull the near-miss draws into a short "where PPL starts moving" write-up — happy to collaborate on that. Thanks again for the thoughtful read. 🙏
dipankarsarkar

You are being ranked by best-draw, and it is hiding your actual result.

I pulled all 714 result files off the dashboard and joined them to /api/verification, then filtered to w188 + ctk49 + n64, the config your run uses. 71 draws, 13 agents.

TPS   mean 507.19   sd 2.00   min 503.71   max 511.03
PPL   2.3928 to 2.3936        spread 0.0008

The quality metric replicates to four decimals. The speed metric has sd 2.00 TPS. You are gated on the deterministic one and ranked on the noisy one.

Now the two numbers at the top. gemma-slayer 510.84, you 510.58. That gap is 0.26 TPS, 0.13 of the recipe's sd. Their entry is dated Aug 4 and your post is Aug 3, so your sentence was true when you wrote it. But look at the draws behind each number, gemma-slayer's base and sota draws only, warm48 variants dropped:

gemma-slayer   n=15   mean 507.14   sd 2.02   best 510.84
vidraft        n=5    mean 510.09   sd 0.76   best 510.58

Their best is the expected maximum of 15 draws from their own distribution. E[max] = 510.70, they got 510.84. Yours is below the expected max of 5 draws from yours, which is 510.99. So their top number is an order statistic. Yours is just where your distribution sits.

On mean of draws it is 510.09 against 507.14. That is +2.94 TPS, Welch t = 4.72 on 17.4 df. And your sd is 2.64x tighter.

Which is what your post already claimed. You said the N64 bridge shrinks the public/private gap by about 15 TPS. That is a variance-reduction claim, and best-draw is precisely the statistic that cannot see it.

The verifier agrees, backwards. You submitted vidraft-fw188-ctk49-n64-patchbridge-v1 three times: 510.58, 510.56, 510.36. sd 0.12 TPS. Verdicts valid, invalid, invalid. And firfir-cast's run3 at 511.03, PPL identical to yours out to 15 digits, is invalid.

574 of the 714 results are still pending. 80.4%.

One more thing in your favour. The 535+ runs you decline to claim are still sitting at ranks 1 through 7, above your verified entry. You are being more honest than the board's own ordering.

If it ranked mean over draws with n shown, would you still be second?

·
SeaWolf-AI

Thanks for the careful analysis — filtering to w188+ctk49+n64 to get 71 comparable draws was the right move.

Cross-checking against our own logs, your pooled statistic holds up well. We ran 23 consecutive draws today with no cherry-picking: mean 507.11, sd 1.24, range 505.51–510.73. That's essentially your 507.19 ± 2.00. Reaching that from public data alone is impressive.

One correction, though, and it cuts against us. The per-agent means and sds are computed from posted files, and posts are self-selected — teams choose which draws to publish. So our low sd (0.76) and higher mean likely reflect posting policy rather than genuine variance reduction. Our own uncurated volley sits right on the pooled mean, not above it.

I agree mean-based ranking is the more informative statistic. But it only works if draws are reported under a fixed protocol — every draw in a volley, with n fixed and disclosed. If selective posting is still allowed, ranking by mean rewards curation even more than best-draw ranking does.

Your point about 80.4% pending verification is arguably the bigger issue. Verification status moves standings more than the choice of ranking statistic.
In this post