The per-benchmark rigor is the easy part and you nailed it: bootstrap CIs, rank ranges from paired items, chance-normalization. The composite is where the order actually gets set, and it carries a second source of variance you don't cost.

50/25/25 is a choice, not a measurement. Two models can swap composite rank from the weight vector alone, no capability change. Your rank ranges propagate sampling noise, not weight-sensitivity.

So a rank stable under bootstrap resampling can still flip under any other defensible weighting. The interval you protect is not the number that sets the leaderboard order.

Do the rank ranges survive perturbing the weights, say 40/30/30 or equal thirds? Or is composite rank calibrated against sampling only, not against the weighting?

The reason you gave for 50/25/25 is stronger than the number, and it points at a different robustness check than the one you proposed.

You did not defend the weights as a capability judgment. You defended them as a correction for what the benchmark set happens to contain: three of five tasks are commonsense, so equal weighting hands commonsense 60%. That makes the weight vector a function of the task list, not a claim about which capability matters more.

If that is the rule, 40/30/30 and equal thirds are not the interesting alternatives. They are other points on the same axis. The alternative that tests your actual claim is to hold category balance fixed and vary which benchmarks fill each category. If ranks move under that, the composite is measuring your benchmark selection rather than the model.

Your expansion plan makes this urgent instead of academic. You said the weighting gets revised as benchmarks are added. So a model's composite rank can change with no re-evaluation of that model, and two ranks recorded under different weight vectors are not comparable, even though the leaderboard prints them in one column.

One smaller thing on ARC-C. Chance-normalization divides by (1 - chance), which scales that benchmark's variance by the same factor. A near-chance task comes out of normalization noisier than it went in, and then gets downweighted at the category level for being noisy. Two corrections for one problem, and the second one also lands on the well-behaved tasks sharing that category.

Does each result row carry the weight vector it was scored under? Without that stamp the new rank range tells you today's sensitivity, but the history still reads as a single ranking.

Versioning the suite instead of stamping the row solves it inside the leaderboard and leaves it broken everywhere else.

A rank that gets cited, screenshotted or quoted in a model card travels without the version. Superseding keeps your live board internally consistent, but the number in the wild is still naked. The stamp is one field in the row and the API output, and it costs you nothing.

On the robustness check, I would push back on the acceptance rule. You said you want to test whether it produces useful additional information before making it methodology. The outcome that produces the least visible information is the one that matters most: if ranks hold across benchmark substitution and leave one out, that is the result that licenses the composite. Ship it only when it moves ranks and you have made publication conditional on the check being embarrassing, so it never ships.

The ARC-C clarification is the useful part and it settles more than you meant it to. 25% science, 75/25 inside it, so ARC-C is 6.25% of the composite. At that weight it cannot move a rank much no matter what it says. It is already diagnostic. The open question is not whether to demote it.

The real issue is that the rule generalizes. Near chance means noisy means lower internal weight. That is not a fact about ARC-C, it is a fact about any hard task, and hard tasks are exactly the ones that separate the top of the ladder. As the suite expands, the discriminating benchmarks are the ones your own weighting rule pushes toward 6%.

So when you add the next benchmark and most of the ladder sits near chance on it, what sets its weight?