Add final information-metric audit

evaluation/controlled_d10_100k_10k/2026-08-17/METRIC_AUDIT.md

@@ -0,0 +1,153 @@
+# Post-evaluation information-metric audit
+
+Date: 2026-08-17
+
+## Bottom line
+
+The first four-tokenizer cohort does not support the intended claim that a
+standard information-theoretic summary predicts controlled gFID. Conditional
+rate, marginal entropy, position entropy, nominal capacity, entropy savings,
+rate/utility efficiency, profile mismatch, and expected total corruption damage
+all fail to recover the ordering
+
+`VAR-4K < MSVR-8K < MSVR-16K < MSVR-4K`.
+
+One exploratory tokenizer-only statistic does recover that ordering: the shape
+of the uniform exactly-one-symbol sensitivity profile. The result repeats over
+three perturbation seeds, but it is not invariant to the corruption policy and
+was discovered after inspecting gFID. It is a candidate for an independent,
+preregistered test—not evidence that the original entropy/rate direction works.
+
+## Outcome table
+
+All values are lower-is-better except where noted. Published rFID is contextual
+and was not recomputed in the controlled local pipeline.
+
+| Tokenizer | Published rFID | Controlled 10k gFID | Probe bits/image | Pooled marginal entropy bits/image | Final LPIPS |
+|---|---:|---:|---:|---:|---:|
+| VAR-4K | 0.90 | **25.218** | 6,761.0 | 7,965.2 | **0.1807** |
+| MSVR-4K | 0.80 | 30.761 | **5,477.8** | **6,801.0** | 0.2211 |
+| MSVR-8K | 0.70 | **27.214** | 5,978.6 | 7,348.5 | 0.2069 |
+| MSVR-16K | **0.67** | 28.512 | 6,163.7 | 7,858.5 | 0.2240 |
+
+The conventional information quantities are especially unconvincing across
+families. Probe bits/image has Pearson `r=-0.928` and Spearman `rho=-0.8` with
+gFID: the best generator condition, VAR, has the greatest total conditional
+burden. Pooled and position entropy are similarly inversely ordered. Final
+LPIPS (`r=0.878`, `rho=0.8`) is more aligned with gFID than any scalar rate or
+entropy quantity tested, but still swaps MSVR-4K and MSVR-16K.
+
+## Metrics audited
+
+The audit reconstructed the complete per-scale profiles and tested the
+following interpretable summaries without fitting a regression:
+
+- total conditional probe bits/image and bits/symbol;
+- nominal capacity and pooled/position marginal entropy per image;
+- `marginal_entropy - conditional_rate` and normalized compression ratios;
+- centers, normalized entropies, and coarse/late fractions of the rate and
+  entropy profiles;
+- total LPIPS/MSE prefix gain and conditional bits per unit prefix gain;
+- utility-weighted rate, Jensen-Shannon divergence, and Wasserstein distance
+  between scale-wise utility and rate allocation;
+- total one-symbol and fixed-fraction damage under uniform and nearest-code
+  corruption;
+- additive expected damage using held-out per-scale top-1 error rates.
+
+None of these quantities exactly ranks all four conditions. Testing more
+algebraic combinations on four observations would be metric fitting, not
+scientific evidence.
+
+## Reproducible sensitivity-shape lead
+
+Let `S_k` be mean reconstruction-MSE damage after replacing exactly one atomic
+symbol at level `k` with a uniformly selected wrong symbol. Normalize damage
+across the hierarchy:
+
+```text
+q_k = S_k / sum_j S_j.
+```
+
+Two related summaries were evaluated:
+
+```text
+H_S = -sum_k q_k log2(q_k) / log2(K)
+C_S = sum_k q_k * (r_k - r_min) / (r_max - r_min),
+```
+
+where `r_k` is the spatial side length of level `k`. `H_S` is the normalized
+entropy of damage allocation. `C_S` is a resolution-weighted center and avoids
+treating MSVR's two separate 1x1 levels as two different spatial scales. Lower
+values mean vulnerability is concentrated in fewer, coarser levels rather than
+spread through the hierarchy.
+
+The same class-balanced 2,000 images were evaluated with perturbation seeds
+1234, 4321, and 9876:
+
+| Tokenizer | gFID | Mean `H_S` ± seed SD | Mean `C_S` ± seed SD |
+|---|---:|---:|---:|
+| VAR-4K | **25.218** | **0.3340 ± 0.0045** | **0.02821 ± 0.00056** |
+| MSVR-4K | 30.761 | 0.5622 ± 0.0047 | 0.03585 ± 0.00052 |
+| MSVR-8K | **27.214** | **0.5387 ± 0.0033** | **0.03157 ± 0.00077** |
+| MSVR-16K | 28.512 | 0.5525 ± 0.0050 | 0.03421 ± 0.00068 |
+
+Both metrics have Spearman `rho=1.0` with gFID for every perturbation seed.
+Across the three seeds, Pearson correlation ranges from `0.822` to `0.831` for
+`H_S` and from `0.972` to `0.985` for `C_S`.
+
+This is substantially more robust than the previous mixed error-amplification
+proxy because it is tokenizer-only, uses one prespecified corruption policy,
+does not use generator error rates, and includes VAR. It also survives a fresh
+corruption draw rather than relying on the original seed.
+
+## Why this is not yet a positive result
+
+1. There are only four outcomes, and this metric was selected after examining
+   them. Correlation significance is not meaningful after this search.
+2. All three perturbation seeds use the same 2,000 validation images. They test
+   corruption Monte Carlo noise, not dataset-sampling uncertainty.
+3. Uniform wrong-code corruption is not the generator's empirical error
+   distribution. Nearest-code sensitivity gets only `rho=0.8` and consistently
+   swaps MSVR-4K and MSVR-16K.
+4. Fixed-fraction corruption is supportive for entropy but not perfectly stable
+   for the resolution center.
+5. `H_S` is sensitive to how a tokenizer enumerates duplicate-resolution
+   levels. `C_S` reduces that problem but normalizes each hierarchy by its own
+   terminal resolution.
+6. VAR and MSVR still differ in hierarchy, tokenizer family, symbol count, and
+   generator FLOPs. The exact ordering could reflect those family differences.
+
+Most importantly, `H_S` is an entropy of a robustness profile, not code entropy
+or conditional source rate. If it generalizes, the scientific direction is
+better described as **hierarchical error localization and propagation** than as
+a generic information-theory predictor of generation quality.
+
+## Recommended decision gate
+
+Treat the current study as a negative pilot for the original entropy/rate
+hypothesis. Do not optimize a composite score or commission 50k FID runs yet.
+
+Before selecting any additional results, preregister exactly:
+
+1. primary metric: uniform one-symbol normalized-resolution center `C_S`;
+2. secondary metric: normalized sensitivity entropy `H_S`;
+3. negative-control policy: nearest-code one-symbol sensitivity;
+4. target: rank controlled gFID on an independently selected cohort of at least
+   six additional hierarchical tokenizers;
+5. pass criterion chosen before training, including required rank correlation
+   and improvement over rFID/final LPIPS baselines.
+
+If that independent test fails, close or substantially pivot this direction. If
+it succeeds, rerun publication-grade 50k gFID and collect the generator's actual
+wrong-code confusion distribution to test the error-propagation mechanism.
+
+## Artifacts
+
+The original Day-1 profiles are under `outputs/info_theory/*_2k.json`. Repeated
+raw profiles are under `outputs/info_theory/robustness/` and are archived at:
+
+<https://huggingface.co/iskhare/info-theory/tree/main/evaluation/controlled_d10_100k_10k/2026-08-17/robustness_repeats>
+
+The repeated files use the identical stratified subset (`subset_seed=1234`) and
+perturbation seeds 4321 and 9876. The 4321 run includes uniform and nearest-code
+policies; the 9876 run is uniform-only.

evaluation/controlled_d10_100k_10k/2026-08-17/REPORT.md

@@ -9,25 +9,33 @@ Status: controlled 100k-step generator evaluation complete; publication-grade
 
 ## Executive finding
 
-The hypothesis receives **qualitative, partial support** in this first cohort.
-Reconstruction quality alone does not determine downstream generation quality.
+The original entropy/rate hypothesis is **not supported as a predictive result**
+by this first cohort. Reconstruction quality alone does not determine downstream
+generation quality, but neither conditional rate, marginal entropy, entropy
+savings, rate/utility efficiency, nor expected total error damage explains the
+four-tokenizer gFID ordering. A post-evaluation audit found one reproducible
+tokenizer-only lead—the scale distribution of uniform one-symbol damage—but it
+is corruption-policy dependent and was selected after observing the outcomes.
+It should be treated as a preregistered follow-up candidate, not as confirmation.
+
+The detailed metric audit is in `docs/info_theory_metric_audit.md`.
+
 The official XQ-GAN model zoo reports monotonically improving MSVR rFID as the
 codebook grows (`0.80 -> 0.70 -> 0.67`), while the matched controlled VAR-d10
 screening gFID is non-monotonic:
 
 | Condition | Published rFID | Controlled gFID (10k) | Rank |
 |---|---:|---:|---:|
-| VAR-4K | reference only | **25.218** | 1 |
+| VAR-4K | 0.90 (XQ-GAN reference) | **25.218** | 1 |
 | MSVR-4K | 0.80 | 30.761 | 4 |
 | MSVR-8K | 0.70 | **27.214** | 2 |
 | MSVR-16K | 0.67 | 28.512 | 3 |
 
 Within the controlled MSVR family, 8K is the best operating point. Moving from
-4K to 8K appears to gain enough representation fidelity to outweigh the added
-prediction burden. Moving from 8K to 16K yields only a small published rFID gain
-but adds 185.1 probe bits/image and worsens screening gFID by 1.297. This is
-consistent with a fidelity-versus-modelability tradeoff, not with a claim that
-any single measured statistic is already a sufficient predictor.
+8K to 16K adds 185.1 probe bits/image and worsens screening gFID by 1.297 despite
+the small published rFID gain. This is compatible with a fidelity-versus-
+modelability tradeoff, but the measured rate-distortion summaries do not predict
+the complete ordering and therefore do not establish that mechanism.
 
 VAR-4K is best in this run, but it is a separate tokenizer family with 680
 atomic symbols/image rather than 572, substantially greater generator FLOPs,
@@ -129,11 +137,10 @@ The summed exactly-one-symbol sensitivity damage is:
 
 A rough additive uniform-error proxy,
 `sum_k L_k * (1 - top1_k) * S_k^(1)`, is U-shaped within MSVR
-(`0.0825, 0.0790, 0.0846`) and matches its gFID ordering. The corresponding
-nearest-code proxy does not. This is suggestive only: independence/additivity
-is not established, and top-1 error rate does not identify the generator's
-actual wrong-code distribution. It motivates measuring empirical confusion-
-weighted sensitivity rather than selecting a corruption policy post hoc.
+(`0.0825, 0.0790, 0.0846`) but swaps MSVR-4K and MSVR-16K relative to gFID and
+fails badly on VAR. The corresponding nearest-code proxy also fails. Top-1
+error rate does not identify the generator's actual wrong-code distribution,
+so neither proxy is evidence for the proposed amplification mechanism.
 
 ## What the experiment supports
 
@@ -141,15 +148,17 @@ weighted sensitivity rather than selecting a corruption policy post hoc.
    rFID trend does not yield monotonic controlled gFID.
 2. **Conditional rate is also insufficient by itself.** MSVR-4K has the lowest
    probe bits/image but the worst gFID.
-3. **An allocation tradeoff is plausible.** The MSVR-8K midpoint balances the
-   observed representation and prediction burdens best in this experiment.
-4. **Scale-local robustness may help explain amplification, but the perturbation
-   distribution matters.** The uniform additive proxy agrees with gFID; the
-   nearest-code proxy does not.
+3. **The tested rate-allocation summaries do not rescue the hypothesis.** Their
+   descriptive ordering is no better than the reconstruction diagnostics.
+4. **Hierarchical sensitivity shape is a lead, not a result.** Uniform
+   one-symbol damage concentration reproduces the four-way ordering across
+   three perturbation seeds, while nearest-code variants do not reproduce the
+   MSVR-4K/MSVR-16K ordering.
 
 This four-point cohort is too small to fit or validate a composite predictor.
-Doing so now would be post-hoc overfitting. The result supports continuing the
-hypothesis, not declaring it established.
+Doing so now would be post-hoc overfitting. The appropriate conclusion is a
+negative result for the original entropy/rate claim and, at most, one narrowly
+defined robustness statistic worth testing on an independent cohort.
 
 ## Limitations and next decisive runs
 
@@ -167,11 +176,13 @@ hypothesis, not declaring it established.
   environment and checkpoints loaded strictly, but the environment difference
   is recorded.
 
-The next priority is a 50k evaluation of all four conditions with exactly 50
-samples/class. Then record the generator's empirical scale-specific wrong-code
-confusions and evaluate `Pr(error type at k) * damage(error type at k)`. A
-matched local rFID rerun and an independently chosen follow-up tokenizer cohort
-would make the mechanism test substantially stronger.
+Do not spend on a 50k evaluation merely to strengthen the current metric claim:
+the two deterministic 5k halves already preserve the screening rank. First
+preregister the sensitivity statistic and test it on an independently selected
+hierarchical-tokenizer cohort. Only if that test succeeds should the four
+conditions receive publication-grade 50k evaluation. Recording the generator's
+empirical scale-specific wrong-code confusions remains the correct mechanistic
+test of `Pr(error type at k) * damage(error type at k)`.
 
 ## Reproduction and artifacts