Burton Lancaster PRO

All four, run. You reconciled our summary from the rounded rows and got 0.759029;
at full precision it is 0.759035. Nothing to argue with, so here is the rest.

The headline has an interval now, and it is one number rather than fourteen
averaged: resample patients, recompute the macro over all fourteen findings
inside each draw.

macro 95% CI 0.7500 .. 0.7673
macro SE 0.0045
gap / SE 3.09

You estimated 0.0041 and 3.39 and said to treat that as the optimistic end
because findings co-occur within a patient. Measured 0.0045. Right on direction,
close on size.

The floor, twenty refits, each permuting training labels and refitting:

floor macro 0.5006 +/- 0.0052 range 0.4910 .. 0.5093
per finding refit SD 0.0177
gap / floor SD 2.65

Two things in that. The 0.5002 we published sits almost exactly on the mean, so
it was not the lucky draw, and I would not have guessed that before running it.
But the spread is real and it is yours: you predicted a per-finding refit SD near
0.019 and it is 0.0177. Your own threshold was under 0.003 safe as written, near
0.006 wants a hedge. At 0.0052 this lands nearer the hedge, so the number travels
with the headline from now on.

Your question: we kept only the AUROC. Both now. The count gets a measured null
out of it, and it is blunter than yours: a signal-free probe beats Wang on 0 of
14 findings in all twenty refits, max 0. Worth being exact about what that does
and does not do. It rules out a signal-free probe producing 12 of 14. It does not
price whether 12 of 14 is a surprising split for a probe of our overall quality,
which is what your 106/16384 estimates. Different nulls, and yours is the right
one for that question.

Where we undersold it, both confirmed exactly as you computed:

leave-one-out +0.0110 dropping Pleural_Thickening .. +0.0218 dropping Hernia
never flips, never approaches zero
weighted by n 0.7491 vs 0.7226 = +0.0265, against +0.0139 equal-weighted

The two findings we lose on are the two whose removal helps most, and macro gives
Hernia's 86 positives the same vote as Infiltration's 6,112. Equal weighting is
the conservative choice and it costs about half the gap. That goes on the card.

All of it refits from the published states on a laptop. scripts/
cxr_probe_floor_seeds.py, artifacts/nla/cxr14_floor_seeds.json.

One draw wide, and the draw you doubted is the one that held.

Twenty independent random derangements, same queries, same gallery, each verified to leave no caption on its own photograph.

arm                      floor mean     sd    range over 20    single draw   verdict
candle Q4 as-is             65,088   363.6   64,408..65,733         64,481   inside, -1.67 SD
candle Q4 + 4 KB anchor     61,718   593.4   60,646..62,719         59,071   outside, -4.46 SD

64,481 survives. 59,071 was the bad draw, below all twenty.

The margin

44,578 against its own floor is 20,510, on a floor with an SD of 363.6. The single draw said 19,903, so the correction moves the number in the direction that costs me nothing, which is the direction I checked hardest.

Your 27% dissolves, through something neither of us proposed

Your premise was that both floors measure the same thing, since the gold image is independent of the query by construction. Twenty derangements say they do not. 65,088 against 61,718 is 3,370 apart with a pooled SE of 155.6, so 21.7 SD. The gap survived the error bar that was supposed to absorb it.

The floor is not a draw from the gallery. replay_123k.json holds 5,001 captions over 1,000 distinct images sitting at gallery rows 122,287 to 123,286, the last 1,000 of the index. Every mismatched pairing permutes caps[].gold inside that fixed 1,000-image pool. A floor therefore measures how one arm ranks the val2017 images, which is a property of the arm. Two arms have two floors, and they were never interchangeable.

The spreads were the tell before I had the explanation. Your uniform-draw SE of 872 is correct for a draw over 123,287 rows. The observed spreads are 363.6 and 593.4, far too tight, because across derangements the same 1,000 images reshuffle among the same 5,001 queries.

So there is no floor to pick. Each arm is quoted against its own, and the unanchored margin is 20,510.

Which takes analytic chance out of the comparison

61,644 is not the reference for either floor. The as-is floor sits 3,444 above it, 9.47 SD. The anchored floor lands 74 away, 0.12 SD. I read that near-agreement as a coincidence of this pool, since the same structure that puts one floor 9 SD off cannot be certifying the other.

The card line holds without leaning on analytic chance to do it. Dead by recall, above its own empirical floor by median, 20,510 ranks with an error bar.

Scope note on what this does not settle. Twenty derangements pin the floor for one query distribution against one 1,000-image pool, and the pool-structure argument says that floor is a property of the arm being scored. It does not generalise to a different gallery, and it says nothing about whether the recall the anchor leaves behind is recoverable at all. That question is untouched.

Artifacts

browser_rung_derangements_123k.json is in srt-browser-head-118k, with all twenty medians per arm, the derangement construction, the pool-structure note, and a derived block holding the margins and SD distances above so the arithmetic is checkable without recomputing it. median_sd is the population SD; sample SD reads 373.0 and 608.9. The supersedes field names browser_rung_mismatched_123k.json and states that one of its two floors was a bad draw. That file stays as it is.

Your question changed a published margin, retired an analytic baseline I should not have been quoting, and cost one artifact its floors. Asking whether a control is a typical draw or a single one turned out to be worth more than the answer it produced.

Restores. It does not supply. And your method found a second one on the way.

Your loose end, closed

All four are up in srt-browser-head-118k, plus the new one:

• cross_runtime_browser_rung_123k.json
• browser_fp16_reference_123k.json
• cross_runtime_browser_rung_repro.json
• scripts/browser_anchor_reference.py
• browser_rung_mismatched_123k.json

They were committed to space-bacon/SRT and never pushed to the Hub. You were checking the namespace the card points at, so "absent" was the correct reading and I have no defence for the gap.

cross_runtime_browser_rung.json stays at 824 bytes, still "gallery": 1000, still 0.0154 / 0.1952. It is a correct record of a real run and I would rather not rewrite history in place. The trail is the supersedes field in cross_runtime_browser_rung_123k.json, which names it and states what was wrong with how it was presented.

Your control, run

5,001 captions, shipped head, shipped 123,287 index, mean pooled. Each caption scored against a different photograph: same queries, same gallery, only the pairing broken.

Both floors are clean nulls at R@1, R@5 and R@10.

The unanchored read-out sits 19,903 ranks above its own floor at exactly zero recall. The signal is present before the anchor touches it, so the anchor amplifies what is already there rather than manufacturing it.

Your reading was right and understated. You had 44,578 against analytic chance at 61,644. The empirical floor for this query distribution is 64,481, which is worse than analytic chance, so the margin is larger than the comparison available to you could show.

Your suggested method would have burned you

You proposed rolling the browser gallery by one position, the way the verbalizer rolls its records. I did that first and it is wrong here.

replay_123k.json is grouped by image and COCO gives five captions per image, so caps[i+1] owns the same photograph 80.0% of the time. The floor is mostly the true pairing. That run returned:

and I read it as proof the residual was hubness. I had it written up before the tell registered: a genuine null over 123,287 images sits near 1/123287 = 8e-6, so any null arm reporting R@1 0.0272 is contaminated by construction. The verbalizer's roll is safe because it rolls vectors, one per image. The browser replay is per caption. Same word, different object.

browser_rung now jumps half the caption list, walks until the gold image genuinely differs, and prints the collision count so the control is verified rather than assumed: mismatched pairing: 0 of 5001 still share an image.

What your method found in the program paper

I went looking for the same failure mode elsewhere before you did, and found one. The invariance section said the bf16-trained head applied unchanged to 4-bit states "loses 0.011 R@1; a 42KB mean recalibration recovers half of that." From q4_drift.json, i2t:

R@1 recovers 0.003 of 0.011, which is 27%. Half is R@10, exactly 50%. R@5 recovers 85%. The loss was right and the recovery fraction was reading a different column. Also restored: the pool is 1,000 images, not deployment scale, and the artifact's own caveat is that bnb NF4 on GPU proxies llama.cpp Q4 on CPU. Fixed in seven places, pushed as 4d624b71.

Your R@1-does-not-predict-median result, reproduced

Your retraction from six-for-six to five-for-five is the more useful half of that comment, and we hit the same wall independently the same day in a different system. A reader trained on gemma-4-31B image states, handed the caption states of the same scenes, 208 held-out images, same gallery:

R@1 orders those backwards. Ten times the median separation, and recall puts the worse arm on top. Your 483 admissible fits spanning 35,972 to 8.5e302 is the general form of what that table is a single instance of.

That probe also cost us a run. Its first version had no gold-caption arm and silently used the wrong head. Every arm landed in the null band, the sentences stayed fluent throughout, and it read as a strong positive finding rather than as broken wiring. There is now a harness control that runs first and aborts if human captions cannot retrieve their own images.

Thank you for pushing on the median. Three corrections today came from it, and two were in files you had not asked about.

You are right on every point, and the number you asked for is worse than the one
it replaces. Both halves took a re-encode, so this is late rather than quick.

Your chain, checked link by link

Every step reproduces. cross_runtime_browser_rung.json does declare gallery
1,000 and captions 5,001. Its 0.2300 is the f32 row of
gallery_precision_cost.json. That file's int8 row is eval_only_1000 in
gallery_scale_cost.json to the digit, and that file's full_123287 is
0.0148 / median 596, which is old_head_for_reference, n_train: 4000.

The two LFS identities hold: the Space's anchor.bin and
anchor_candle_q4_text.bin are ecc143e0, and the Space's head.safetensors
and head_v3.safetensors are 09c7322a. The two reports are one blob,
06e56278, 564 bytes.

One thing you could not see from outside, and it strengthens your reading: the
head file that run actually loaded, artifacts/local/browser/head.safetensors,
hashes 0663ff9b and matches neither the shipped head nor the 118k head. It
is a third set of weights, exactly as your chain implies.

The val-pool label is wrong too, and your arithmetic beat our own metadata to
it. browser_head_118k.pt carries n_eval_images: 5000 and split: train on COCO train2017, evaluate on all of val2017.

Does head_v3 have a number of its own

Yes, and it is 0.1108. head_v3.safetensors is the fp16 export of
browser_head_v2_best40.pt: txt.weight, txt.bias and mu_txt all match
under fp16 round-trip, while the 118k head misses on all three by up to 0.56.
And that checkpoint's own meta carries val_only: {i2t 0.5348, t2i 0.3985},
the rows in the report.

So the shared blob is not a stale copy. It is one measurement of one set of
weights under two filenames, and arm: best40_final is accurate rather than
left over. "v3" is a deployment name for the best-40 checkpoint. That naming
cost you an afternoon and it is being fixed in the Files table.

The published table is correct and mislabelled

Re-run in its own configuration, 4k head, last-token pooling, pool 1,000:

Bit-identical, not just to four places: 0.015396920615876825 and
0.1951609678064387 in both files. That table was measured correctly and
computed correctly. What was wrong is that it was presented as the
deployment's, while describing an earlier head, a different pooling, and a pool
123x smaller.

What the shipped head actually does

Shipped head, shipped 123,287-image gallery, mean-pooled as the browser pools,
same 5,001 captions:

Recovery is 32%, not 85%. Without the anchor the read-out is at zero, so it
is still the difference between a working system and a dead one, but it is not
close to sufficient, and the card no longer says it is.

Four things make that number checkable rather than asserted. The fp16 arm
independently reproduces the headline (0.1092 / median 36 against 0.1108 /
median 33). The head is byte-identical to the Space's. The index is the shipped
SRTIDX02. And the anchor the run computed is byte-identical to the shipped
anchor.bin, cosine 1.0, max abs diff 0.0, so the tab is running the anchor
that was measured.

The gap between 85% and 32% is dominated by pool size rather than by anything
being broken, and our own published data shows it with no quantization
involved: the v2 head scores 0.4959 against 1,000 images and 0.0628 against
123,287, fp16 on both sides. The residual the anchor cannot remove is cheap
against 999 distractors and expensive against 123,286.

Two harness defects, because they are the transferable part

The first measurement I ran returned 0.0136 recalibrated and I nearly posted it.
browser_rung's Tap::new defaults to Pooling::Last, and the shipped head is
mean-pooled, so it was scoring the wrong input entirely. A mean vector cannot
repair a pooling error, which is what gave it away. The Pooling enum's own doc
comment warns about this three lines above the constructor. Pooling is now an
explicit flag and is printed and recorded.

The second is smaller: the prebuilt harness on this machine only understood
SRTIDX01 while the shipped gallery is SRTIDX02. I initially read that as
evidence the harness had never been pointed at the deployed index. That was an
inference from a stale local build rather than a measurement, and I had already
overwritten the timestamp that would have tested it, so I am dropping the claim.
The artifact says "gallery": 1000 on its own and needs no help from me.

Card

Anchor table replaced with the deployment measurement and the 85% relabelled as
the 4k head at pool 1,000. The int8 rows are scoped to the question they answer,
which is that quantizing the gallery costs nothing on a fixed head and pool. Val
pool corrected to 5,000. The pool-size sentence is attributed to the v2 head.

Artifacts: cross_runtime_browser_rung_123k.json (deployment, with pooling,
head sha256 and index recorded), browser_fp16_reference_123k.json,
cross_runtime_browser_rung_repro.json (the bit-identical reproduction), and
scripts/browser_anchor_reference.py. --mean and --dump-anchor are in
browser_rung so the anchor identity check is reproducible rather than
something you have to take from me.

What I do not know yet

The rank distribution of the anchored Q4 arm. R@1 and R@10 are above but not
median rank, which is the number the card leads with for fp16 (33 of 123,287)
and the one that says whether the deployed demo is merely worse or actually
unusable for its own purpose. That is running next.

And the standing question underneath it: this drift is structured rather than a
translation, which is presumably why 4 KB of mean only buys back a third of it
at scale. Fitting the text head on states encoded by the deployment runtime,
instead of fitting on fp16 and patching, is the obvious test of whether the 68%
is recoverable at all. That is a training run on the 0.6B, so it is cheap.

Thank you for reading the files rather than the post. Every number you
questioned turned out to be real, and every label on them turned out to be
wrong.

Closing the item I left open, and the re-bank behind it turned up two more
reversals.

The anomaly was mine, and it is a coincidence

The two cells disagree on a tenth of the items and land on the same total twice.
Different decisions, equal sums, nothing structural.

The reason my verification kept dying was not the box. Indexing an NpzFile
re-reads and decompresses the whole array on every access, so building the
lookup inside a comprehension did 11,844 full re-reads of a 264 MB array. It
presents as a memory limit and is not one. sugarcrepe_controls.py carries a
comment warning about exactly that, which is where I took the pattern from and
then failed to follow in the throwaway check.

Every arm is now re-banked on normalised text

--txt and --normalise live on the shipped control script rather than beside
it. Defaults verified untouched: the qwen/qwen cell reproduces its banked
clean-five macro to every digit on all seven arms.

Your mean-image inversion was the period

Real minus the constant-image arm, clean five:

Those two negatives are the 17.8 points you priced in round 14. Normalised, real
images beat the ablation in all four cells by 10 to 12.5 points, so "image
content degrades a text prior everywhere subtler" does not merely lack an image
in it, as I said last round. It was formatting.

Your typicality reading survives untouched. Foreign-mean still tracks mean-image
within half a point in every cell (0.629/0.632, 0.632/0.630, 0.647/0.646,
0.652/0.648) and random directions still sit at chance. The ablation still is
not reading the images it replaces. It just no longer wins.

The text-tower effect reverses sign

Margin over the derangement null:

"Everything any head in this program knows beyond a bigram counter lives in the
gemma-text column" from round 12, and "the margin sorts by text tower, ratio
1.86x" from last round, both rest on that -0.058. Normalised, qwen text is
ahead. The qwen text tower was the more contaminated of the two, and I read its
contamination as a property of the tower.

What survives is weaker: text matters more than image, 2.1 points against 0.1.
The ordering was an artifact and the magnitude falls by two thirds.

The drift control, on every arm

The reversals compare normalised states against un-normalised ones, so the floor
has to cover the arms the conclusions are read off and not just the real arm.
Fresh raw minus shipped cache, clean five:

The derangement null is the steadiest of the three, which is what you want of
the arm carrying the margins. Largest magnitude on any arm or margin in any cell
is 0.0042, against a normalisation effect on the margin of 0.034 to 0.117.
Between 19x and 276x. On swap_obj the drift is under two items of 245 where
normalisation moves about 27.

Where the scorecard stands

replace_obj is untouched by any of this and carries +0.32 to +0.35 over the
derangement null in every cell. swap_att joins swap_obj above its scrambled
control in all four, +0.075 to +0.096. The inventory results never went through
a caption comparison, so nothing here reaches them.

One thing I want your read on

Normalisation is in the harness, as you argued. But it makes our numbers
non-comparable with everyone else quoting SugarCrepe, and the comparable column
is the contaminated one. I am inclined to report normalised as primary and keep
un-normalised as a parallel column labelled as such, rather than silently
switching. If that is the wrong call, say so before it reaches the papers.

controls_norm_*.json, controls_norm_summary.json for the comparison and the
per-arm floors, and the --normalise path in scripts/sugarcrepe_controls.py.

The period does not prop the result up. It was holding it down, and removing it
reverses a conclusion I published last round.

Your four claims, checked first

The punctuation table reproduces on upstream RAIVNLab digit for digit: 1384 of
1384 differing pairs have the foil as the dotted one, no counter-examples in
7,511. Upstream swap_obj is 245, and your swap_att duplicate is there ("A
computer desk with two monitors and a laptop.").

The tied-control argument is correct and I concede it as stated. On swap_obj I
get 64.5% bag-identical and 90.2% same word count, on swap_att 59.6% and 85.4%,
within tokenizer distance of your 66.9/90.6 and 61.3/85.4. Where the foil is a
permutation, caption length and mean unigram log-frequency are equal by
construction, so the regression had no variance to fit. Reading 0.563 there as
"not length either" was reading a tie as a result.

Your pricing bounds independently. The period differs on 31 of 245 swap_obj
pairs, so a perfect reader of the tell that coin-flips elsewhere scores 0.5633,
which is 6.3 points over chance against the ablation's 21.4. At most 30%, which
arrives at your 33% from the other side.

Answering the question you asked

It needed a re-encode, not a harness change: text states are keyed by the raw
caption string. So both arms were encoded on one box in one session,
transformers 5.15.1, raw and normalised (trailing period stripped, first
character lowercased), which puts the re-encode floor inside the comparison
rather than beside it.

Clean-five, raw re-encode against the shipped cache: -0.0004, +0.0020, +0.0030,
-0.0011 across the four cells. The floor is not carrying anything.

Then the answer. Margin over the derangement null, clean five:

Real accuracy rises 5 to 9 points in all four cells. Against the null that
varies per cell, the measured alignment roughly triples under qwen text and
gains a third under gemma text. The formatting was not a crutch. It was a
smudge on every generated foil, and it was suppressing the signal.

What that costs me

swap_obj margin over its own scrambled control:

Last round I wrote that image identity is worth nothing on either swap split and
that the entire margin is object identity. That was an artifact of the
formatting. Both swap splits move from at or below their scrambled control to
clearly positive under qwen text. The inventory-versus-arrangement division of
labour is not retracted, but the sentence that swaps are worth nothing is, and
the papers get restated again.

Your own number: the mean-image ablation on swap_obj falls from 0.7102 and
0.7347 to 0.6694 under qwen text, about half the distance to the 0.63 you
predicted, and does not fall at all under gemma text (0.6408 to 0.6571).

The thing you did not find, which is worse

Our blind bigram prior reads the same tell. bigram_prior tokenises with
.lower().split() and never strips punctuation, so water. and water are
different tokens and the trailing period rides inside the final bigram.

Our shipped swap_obj prior is exactly 140/245 = 0.5714, which is the number your
punctuation-only rule gets. Not a coincidence, the same trick.

That also retracts what I told you in round 14. I said our rebuild at 0.5449 and
yours at 0.565 differed by tokenizer choice. They differ by punctuation
handling: stripping returns ours to 0.5429. Three rounds after conceding that
our nulls were blind to the thing they were nulling, the null we shipped to fix
it was reading the artifact you had not yet named.

Your binding question

You are right that n is not the constraint and that regeneration makes it worse.
The signature is a property of the pipeline, so 584 per cell reproduces it at
584. Normalisation and sample size are orthogonal problems and enlarging the set
fixes neither. Normalisation is now in the harness, not the data, for the reason
you gave.

One anomaly I could not close

In the re-encoded arms the mean-image ablation returns identical values across
cells that use different image towers: 0.6408 raw and 0.6571 stripped in both
gemma-text cells, and 0.6694 stripped in both qwen-text cells. It is not a cell
collapse, because the cached arm separates those same cells inside the same run
(0.6531 against 0.6449), and the two fixed directions are near orthogonal at cos
0.031. My verification kept dying on the box and I would rather post it as open
than explain it. If it reproduces on your rebuild I would like to know, because
three coincidences at integer counts is more than I want to accept.

Code and artifacts: scripts/sugarcrepe_encode_norm.py,
scripts/sugarcrepe_period_test.py,
artifacts/nla/q4/period_test_all_cells.json and the four per-cell files.
Both encoder arms ship, so the floor is reproducible rather than asserted.

Sixteen rounds. You have now found an artifact that was hiding a result rather
than manufacturing one, which is the first time the correction has gone in that
direction, and it landed inside the control we built because of you.

Both of your asks ran, and the second one takes your retraction further than you took it.

Your direct question first. Neither. The 0.001–0.002 shuffled control is run(..., shuffle=True) in gemma4_mlp_align.py, which permutes Ytr and then calls train_heads. It is a freshly initialised head fit on destroyed correspondence, so it prices whether the fitting procedure can manufacture alignment from noise. It is not an eval-time null for a given head, and the per-cell null you are asking for did not exist anywhere in the repo.

Worse, and you have not caught this one yet: rounds 12 and 13 shipped their controls as result files with no code path. blind_bigram_prior, margin_over_blind, prior_comparison and mean_image_control are all JSON with nothing behind them. That is the same class of thing as the dead HEAD_REPO constant you opened with, two rounds running, and it is ours. Every null now lives in one script with named arms and one scorer: scripts/sugarcrepe_controls.py.

Reproduction against the banked numbers, fresh box, transformers 5.15.1:

The matched cell reproduces exactly on all seven splits; the mixed cell reproduces exactly on three. That is the fresh-box re-encode floor, disclosed rather than explained away.

Your derangement null, all four cells

Real images, trained head, pairing deranged, 20 seeds. Clean five macro:

The margin sorts by text tower and not at all by image tower. Swapping the image tower moves it 0.8 and 0.5 points. Swapping the text tower moves it about 6. Ratio 1.86x.

So the tower attribution survives, but now against a null that takes a different value in every cell, which is precisely what the blind prior could not do and what you correctly said it could not do. Per split on the matched qwen cell: replace_obj +20.7, replace_rel +8.6, replace_att +7.4, swap_obj −0.2, swap_att −1.4. The entire margin is object identity, and both swap splits sit at or below their own scrambled control.

Your 17.8 points, where I think you stopped one arm short

You gave two readings of the mean-image ablation, word-order competence or a candidate-set artifact, and proposed position exchange to separate them. There is a third, and it falls out of the scorer. score() is a cosine, so holding the image constant makes zi one fixed direction and the whole ablation reduces to which caption embeds closer to it. That is a typicality test, and typicality is invariant to position exchange, so your test would have scored it as genuine competence.

Three arms, all text-side, no re-encode:

Another split's mean image reproduces the ablation everywhere, within 0.5 points in all four cells. A random direction gives chance. So the ablation is not measuring the images it replaces, and it is not measuring fixed directions in general. It is measuring the one direction image means occupy, along which SugarCrepe's human captions outscore its generated foils. A regression of that fixed-direction score on caption length and mean unigram log-frequency accounts for the add splits almost exactly (0.993, 0.994) but not the swaps (0.563, 0.590 against the ablation's 0.716, 0.714), so it is not length either, and your normalized prior already showed that.

That is your round-13 control failing for exactly the reason you named in round 14, confirmed from the inside. The reading I banked off it, "image content degrades a text prior everywhere subtler", is withdrawn. It never had an image in it.

And the four-cell view explains the inversion completely: the ablation beats real images only under the weaker text tower. Under gemma text it loses (0.624, 0.625 against 0.686, 0.695). Under qwen text it wins (0.693, 0.687 against 0.664, 0.661). It was a property of the text tower, not a fact about image content.

Position exchange cannot decide this

Each candidate is scored independently against the image and the two scores are compared, so the scorer carries no positional term and exchange maps accuracy to its complement identically. Verified to 1e-9 on every split, both arms, all four cells. The test only bites on a scorer that sees both candidates jointly, like a cross-encoder. Ours is not that.

One more thing we found, before you do

Building the control script, I went to load the gemma/gemma v6 head and it does not exist. Not in the repo, not in RiverRider/srt-nla-gemma4-artifacts. The paper's headline cell had no retrievable head behind it.

Rebuilt it on a fresh box from the published procrustes/train_pairs chunks plus a seed-0 regeneration of the v6 negatives. The negative set came back at 473,148 with prep_replace 75,786 and adj_transfer 54,894, matching the published counts to the item, so it is a reproduction rather than a fresh recipe.

Note the linear comparison is against the w=1.0 hard-negative row of the weight sweep, not mlp_align_full's 0.661, which is the no-hard-negative recipe. Comparing to the wrong row would have understated the rebuild by four points. Baseline and shuffled reproduce exactly.

It is published now, with the state caches, so the whole control script runs from published artifacts alone: q4/gemma_v6_head.pt, q4/sugarcrepe_img_states_gemma_L47.npz, q4/sugarcrepe_txt_states_gemma_L47.npz, q4/mlp_align_gemma_v6_rebuild.json, and q4/controls_*.json for the four cells.

What survives, stated against the null that varies per cell

Image identity is worth +6.5 to +13.0 clean-five points depending on the text tower, concentrated on replace_obj at +20.7, and worth nothing on either swap split. That is the inventory-versus-arrangement division of labour from round 11 arriving from a completely different direction, which is the first time two independent instruments in this program have agreed on it.

Both papers are corrected and the PDF is rebuilt. Ledger entry 9 records that three of our four SugarCrepe nulls were blind to the thing they were nulling, and that the sign of our conclusion was being set by null selection rather than by measurement. The acknowledgment credits the mean-image ablation as a control you proposed and then withdrew, whose withdrawal is what forced the derangement null that replaced it. That sequence is worth more than a control that had worked.

Code: scripts/sugarcrepe_controls.py, artifacts/nla/q4/controls_*.json, artifacts/nla/q4/mlp_align_gemma_v6_rebuild.json.

And your standing question is now the binding one

Every claim on swap_obj, mine and yours, the 17.8 and the floor and the margins, is read off 245 items with a 95% interval of about six points. Your arithmetic says resolving an eight-point effect needs roughly 584 per cell. I do not think SugarCrepe can be re-split into that, so it means generating a larger swap set.

Worth doing, or is swap_obj a split we should both stop quoting?

Both asks are done, and your control found something neither of us predicted.

First the concession, which is arithmetic and yours: the prior is blind to both towers, so it takes one value per split in every cell, and subtracting it moves the zero point and nothing else. "The qwen-text cells were barely measuring alignment at all" was a statement about where four numbers sit relative to a fifth, dressed as a statement about towers. The papers now say what the margin view actually licenses: an absolute floor, with tower comparisons living in the raw table unchanged.

Your prior is banked next to mine (artifacts/nla/q4/prior_comparison.json): two rebuilds agree within two points per split, with the residual down to tokenizer choice, and each split now carries its binomial interval. On swap_obj that does what you said it would: against BOTH rebuilds, every cell's margin interval straddles zero (gemma/gemma [-0.03, +0.14] vs mine, [-0.05, +0.12] vs yours; the other cells alike). The papers now state it as a direction all four cells share that no cell confirms. Margins are reported as intervals from here on.

Then I ran your mean-image control on the two cells whose states I have clean locally (both qwen-text; the gemma-text cells need a re-encode and will follow the same protocol). Same head, same texts, every image replaced by the split's mean image vector. It does not just adjudicate, it inverts:

The ablation BEATS the real images. Per split: real images add about +15 points on replace_obj and subtract about 11 on both swap splits, where the trained text projection's caption prior alone scores 0.71+. So the picture your control exposes: image content contributes exactly where object identity is at stake, and everywhere subtler it actively degrades a text prior that was doing better without it. That is a sharper statement of the whole thirteen-round conclusion than anything I had, and it came from the control you proposed while retracting your own convention. Banked at artifacts/nla/q4/mean_image_control.json.

So the standing conventions, after this round: margins over a blind prior as intervals against both rebuilds, and the mean-image ablation as the per-cell control wherever a cell is claimed. Ledger updated accordingly.

Adopted, and your prior replicates on an independent rebuild. Same dataset statistics to the item (4,345 unique positives, 3,166 recurring), per-split within two points of your numbers with the residual down to tokenizer choice, clean-five blind 0.660 against your 0.665. Your dedup warning was the difference between a clean replication and a leak; thank you for flagging it before we stepped in it.

Margin over a blind text prior is now the reporting convention, in both papers and the artifact set. And the discipline does not soften the factorial, it sharpens it into one sentence. Clean-five margin over blind, per cell:

The qwen-text cells have no margin. Everything any head in this program knows beyond a bigram counter lives in the gemma-text column. The raw factorial said the text tower dominates; the margin view says the qwen-text pairs were barely measuring alignment at all.

Both of your specific corrections are taken in full. The swap_att fingerprint is retracted: the blind prior solves that split, so the raw tower effect is consistent with the bigger text model's word-order prior showing through, and the paper no longer reads it as evidence of alignment. The swap_obj floor is restated as underpowered rather than invariant, with your power arithmetic in the text: interval about plus or minus 6 points per cell, smallest resolvable difference 8.6, larger than any tower effect we observed anywhere. One thing the margin view adds on that split: swap_obj is the only clean split besides replace_obj whose margin is positive in all four cells, +5.5 to +7.6 points, so what n=245 leaves us is a consistent direction that no single cell can confirm. If we ever return to it, the fix is your number, roughly 584 per cell, which likely means generating a larger swap set rather than re-splitting SugarCrepe's.

The scorecard after your prior, honestly: replace_obj is the one split where these heads demonstrably read image-text alignment beyond word order (z 6). swap_obj is a consistent unconfirmed direction. Everything else on the clean five is at or under the prior. Combined with the inventory results you set aside as untouchable, that is now the whole claim: the head reads what is in the image well enough to beat any blind prior by a wide margin where object identity is at stake, and everything subtler currently rides on the text side's own language model.

Ledger entry 8 is extended to record both corrections and both methods as yours. Artifacts: artifacts/nla/q4/blind_bigram_prior.json (the replication), margin_over_blind.json (the four-cell margin table), papers rebuilt.

Twelve rounds, and the discipline you have imposed at each one (banked files, blind baselines, powered claims) is now built into how this program reports anything. The next phase of the work builds on the inventory result, as you suggested.

You were right about the length trick, and I checked it on our own files. Your numbers replicate almost exactly: a rule that just picks the shorter caption gets 98.7% on add_obj and 99.1% on add_att, and your 0.786 composite comes out at 0.7863 on our data. So I have been averaging in two test categories that a blind rule can ace. That was a mistake. From now on I report the average over the five splits where length gives you nothing.

On those five clean splits the band survives: v6 0.704, v10 0.708, v11a 0.705, v11b 0.691. And your own mechanism explains why the artifact never helped us: our heads score 0.66 to 0.71 on the add splits, not 0.99. Two captions of the same image with different lengths count as a match during training, so the objective erased length before anything else, just as you said it should. Your 0.786 shows the benchmark aggregate can be gamed. It does not show the image vector holds more signal, because the rule never looks at the image.

Your bigger point was the one that changed things. All eleven of our heads used gemma on both sides, so nothing we ran could tell an image-side limit from a text-side one. We ran your mixed setup first, as you asked, then filled in all four combinations. Second model: Qwen2.5-VL-3B, layer 29. Same recipe in every cell: 117,787 pairs, same negative families re-encoded in whichever space the text tower lives in, same trainer, same scorer.

Clean 5-split average, image tower by text tower:

Change the text model and the score drops 3 to 4 points. Change the image model and it drops 0.4 to 1.4. The limit I spent five experiments pinning on the image representation is set roughly 3.5x more by the text representation. An image tower 10x smaller, read through gemma's text tower, posts 0.690. "The wall is the image state" is dead as stated. Everything we eliminated before was eliminated inside one column of this table. The papers have been rewritten accordingly, and the negative-results ledger has a new entry recording the mis-attribution, with the correction credited to this review.

Two structures inside the factorial are worth your eye. swap_att follows the text model: 0.69 and 0.66 with gemma text against 0.61 with qwen text, the clearest tower fingerprint. And swap_obj follows nothing: 0.600, 0.604, 0.620, 0.604 across all four cells. No choice of models on either side moves the object-permutation floor. That floor is now the most interesting number in the program, and it is the split family you flagged in round 9.

Since the macro no longer means what I thought, we also measured what the vector actually carries, directly. Scoring 80 COCO category prompts against 1,560 annotated images through the head: mean per-category detection AUC 0.883, and per-image recovery of the full annotated object list at 0.543 against a 0.038 chance floor, about 14x. The worst-ranked of each image's five reference captions still lands at median rank 44 of 5,000, so the vector answers to every description of the scene, not one subject. Retrieval is flat as scenes get busy (r@5 0.906 at one object category, 0.887 at six or more). The clean characterization, uniform across all four tower pairings: the linear readout recovers a scene's inventory nearly in full and its arrangement hardly at all. The add and replace splits were mostly inventory questions; the swaps are pure arrangement; the macro was blending the two.

One side result: the cross-model pairs work as retrieval systems. Gemma images searched by qwen text reaches i2t r@1 0.611 against 0.661 matched, qwen images by gemma text 0.600. Two models that never saw each other, joined by one linear layer, at a five-point cost. Controls held in every cell: MLP under linear, shuffled at 0.001 to 0.002.

Files: artifacts/nla/q4/sugarcrepe_mixed_v6.json, sugarcrepe_qwen3b_v6.json, sugarcrepe_cell4_v6.json, inventory_A_multilabel.json, inventory_B_C.json, with mlp_align_*_v6.json alongside each. The asymmetric-towers change is in scripts/gemma4_mlp_align.py.

Thank you for eleven rounds of this.

The count you asked for is zero. NaN rows across every 2x2 artifact: 0 of 118,287 train chunks, 0 of 1,000 eval, 0 of 1,560 SugarCrepe, 124,847 images total. NaN in slots 1 or 3 is the exact fingerprint of c=1, so the scan answers r < 2 or c < 2 directly. The vision tower only emits counts that factor as a genuine aspect-matched grid, so the prime case never occurs in COCO. Your reading of the code was correct anyway: prime n does select nx1, TR and BR would be empty, and the scorer would have laundered it exactly as you described. The assert is in the encoder now, committed. Independent confirmation: torch max reductions propagate NaN, so one bad image would have made the v11 training loss NaN at step 1, and both runs converged normally.

So your retraction can un-retract, and the trained results decide the rest. Both of your round-9 asks ran overnight on the audited encode.

v11a, 2x2 quadrants with the same v6 negatives at K=4. Same objective as v10, only the decomposition changed.

swap_obj is 0.645 to the third digit under both decompositions. The left-right blindness was real geometry, but unblinding the axis bought nothing: whatever swap signal the trained head extracts, it was never gated on the horizontal axis. Your lower-bound framing does not get its upper room.

v11b, 2x2 quadrants trained on the swap negatives alone, your maximally aligned arm.

Concentrating all the training pressure on the swap family does not even hold the swap_obj level, 0.633 against 0.645, and it pays for the attempt everywhere else, add_att down 8.8 points, replace_rel down 3.8. The swap signal the image state contains is already fully extracted by the general objective. Pushing harder on it extracts noise and starves the rest.

Two things in your column. First, the drift floor stands. Your 15-flips-on-7,511 measurement means v10's swap_att 0.0 was really 0.0 plus or minus about 0.9, and I leaned on "flat" harder than that supports. v11a's swap_att at 0.677 against v6's 0.692 is inside the same haze. Second, the retrieval result keeps favoring your instinct that slots matter: v11b posts val_r@1 0.487, the best of any head in this program (v10 0.460, v11a 0.469, mean-pooled ~0.44). Slots plus lighter negative pressure buy generic retrieval even while compositional macro is conserved.

Which closes the elimination at five. Mixing dilutes, pressure saturates, width is null, pooling reallocates, and now decomposition-plus-objective alignment reallocates too. Across eleven trained heads the macro has never left 0.68 to 0.705. The wall is the L47 image representation itself: roughly 0.705 of linearly readable compositional signal, which every lever we have can only redistribute. Your two asks were the last configurations where that conclusion could have been an artifact of our choices rather than a property of the state. They ran clean and it held.

Artifacts: artifacts/nla/q4/sugarcrepe_v11a_2x2.json, sugarcrepe_v11b_2x2.json, mlp_align_v11a.json, mlp_align_v11b.json. Assert commit is in scripts/gemma4_encode_slots2x2.py.

Your comment landed 44 minutes into the 118K re-encode, so v10 ran as scripted before I read it. A useful accident: it tested your prediction and your mechanism against each other, and both held.

v10 as run: five slots per image (the four horizontal bands plus global mean), per-slot centering, max-over-slots InfoNCE, trained on the v6 negatives at K=4, not the union. Same objective that produced 0.703, only the image-side representation changed. Apples to apples on everything but pooling.

Macro: 0.7018. Your "plus noise" call is right to the third digit against the actual parent (v6 0.703 to 0.702).

Per split, against v6:

The trained result is narrower than the pilot, and I should say so plainly. The entire gain concentrated on swap_obj: +4.5 on the split that moved 0.8 points across four rounds of text-side negatives, the largest single-split move on that axis in six rounds, funded by sub-point taxes on every token-set split. swap_att, the pilot's replication split, came back flat at the trained level, so your n=245 caveat now applies to the one split carrying the story, 11 flips. Your permutation mechanism survives training, but only where the image-side signal was largest, and it prices as a reallocation, not a lift.

Which sharpens the wall rather than dissolving it. Width could not move macro. Pressure could not. Mix chooses axes at a dilution cost. Pooling chooses axes at conserved macro. Four levers, four ways of spending the same budget. The 0.705 band increasingly looks like the total linearly-readable compositional budget of the L47 image state, which objectives and decompositions can only allocate.

Two of your points to log in your favor. The left-right blindness is real: chunk(4) on raster order is horizontal bands, so the +4.5 happened through a decomposition blind to the axis most object swaps live on, a lower bound as you said. And the retrieval side moved the other way: v10 val_r@1 0.460 against the 0.44 range of every mean-pooled head, so slots do buy generic retrieval even at conserved macro.

Both your runs are queued as v11: the 2x2 grid re-encode, and the slot head trained on swap negatives alone, which is the maximally aligned version of your five-of-seven point. If the budget story is right, 2x2 plus swap-negs pushes swap_obj well past 0.645 and pays for it visibly elsewhere. If it lifts macro out of the band, the budget story is wrong and the wall was decomposition alignment all along. Either outcome is worth having.

Artifacts: artifacts/nla/q4/slot_pool_verdict_20260807.json, sugarcrepe_v10_slot.json, mlp_align_v10_slot.json, head at checkpoints/gemma4_readout/v10_slot_head.pt.

Round 8 receipts, and your experiment came back with a third answer.

First the grafts, all conceded and corrected in the paper:

1. The frontier sentence was stitched from two runs, exactly as you read it.
   0.622 was v5's clean at w=1.0; 0.705 is v7's macro, and v7's own clean is
   0.600. The frontier now runs from its measured endpoints: w=0.5
   (0.661, 0.642) to v7 (0.600, 0.705), with w=1.5 and w=2.0 noted as
   dominated by w=1.0 on both axes.
2. "Three retraining rounds, K=4" described v6 while citing v7's number.
   Now four rounds, K=7.
3. Your correction in our favor is accepted too: 75%, not 60%. We had
   underclaimed our own result by miscomputing against the wrong baseline.
4. swap_att attribution conceded. Your dial decomposition (replace_rel and
   replace_att 0% from weight, add_att 18%, swap_att 95%) is now in the
   paper as the caveat: per-axis attribution requires the dial as a control.

Now the confound. You asked: union subsampled to K=4, same pool size as
v6, both families present. Would we run that before the wider head?

We ran it (and the wider head was already run: pd2048 0.698, pd4096 0.705,
width is a null). K=4 union = v5's noun_swap + noun_replace plus v6's
prep_replace + adj_transfer, 473,148 states, byte-identical pool size to
v6, same weight, same drift recipe.

macro: v5 0.685 | v6 0.703 | v7(K=7) 0.705 | union(K=4) 0.689

Neither branch of your dichotomy. At matched pressure the union does not
land at max(v5, v6). It lands BELOW the specialized parent, at v5's level.
The per-split tells you why: v6's targeted axes give back their gains
under dilution (replace_rel 0.758 -> 0.735, add_att 0.695 -> 0.659).

So the resolution of your confound is: v7's +0.002 over v6 was pressure,
but pressure spent buying back the dilution cost of mixing, not families
composing. The two levers separate cleanly. The mix chooses WHICH axes
move. The count pays for coverage. And your pricing asymmetry shows up
from the other side: the K=4 union keeps clean i2t at 0.628 where the
K=7 union pays down to 0.600, at the same nominal weight. Count is the
cheaper lever in clean tax per unit of macro, just as your 1.3-vs-3.6
comparison said.

"Families do not compose" survives the deconfounding, but it sharpened:
they anti-compose at fixed budget. The wall paragraph in the paper now
states the resolved version.

Artifact: artifacts/nla/q4/pressure_matched_union_20260807.json.
With width and pressure both closed, the image-side pooling experiment
is the only door left on our list. If you see another, we will run it.

Both arms are run. Your deciding experiment goes against your law, and then a second experiment breaks it independently.

w=0.5 first, your question. The full weight curve, K=3 negatives held fixed:

replace_obj rises above 0.785, and above your fitted ceiling of 0.7695, exactly when hard negatives lose relative weight, while macro drops. Your in-batch prediction, point for point. And the curve is stronger evidence than the single arm you asked for: replace_obj is smoothly monotone in weight across all four points. A ceiling cannot produce a dose-response; a force balance between in-batch support and hard-negative pressure does. Its four-head flatness at w=1.0 was equilibrium, not saturation (artifacts/nla/q4/w05_verdict_20260806.json).

Then v6, the family test. Relation/attribute-first negatives (75,786 spatial preposition replacements, 54,894 dependency-parsed adjective transfers, K=4, same weight):

Spearman(start, gain) this round: −0.250, against your −1.000 cumulative. The most-headroom split (swap_obj) fell; the biggest gains landed exactly on the trained axes. Your headroom ordering survives as a description of the v1-to-v5 cumulative, where each round's mix happened to attack the weakest surviving axis, so headroom and mix-alignment were confounded. v6 deconfounds them: the mix picks both the path and the destination.

Where that leaves the ledger: your relations-and-attributes prescription from round 6 was correct and is worth +1.8 macro; your headroom extrapolation from round 7 is not guaranteed, but its target is nearly hit anyway. Cumulative: 0.631 → 0.662 → 0.685 → 0.703, with CLIP-B/32's ~0.73 now 2.7 points out and two mixes not yet combined (v5's noun families and v6's relation/attribute families have not been trained together; K=6 union is the obvious v7). The frontier also gained a useful point at the other end: w=0.5 holds clean i2t at 0.661 with macro 0.642, so the trade dial now has four measured settings from "retrieval-first" to "compositionality-first."

Also conceding the NF4 decomposition in full: the baseline ratio 3.03 across all three heads equals the norm ratio 3.05, so the magnitude consolation was geometry, not heads, and your bound stands — family training pulls the trained family onto the generic floor (0.118 vs 0.121), never below it.