gliner-tier-probe

Tier probe: a frozen rafmacalaba/gliner_datause_tiered encoder with a small MLP head classifying the tier boundary — T1∪T2 keep vs T3/junk drop. The GLiNER extractor owns the mention boundary only; this head tests whether its representation can expel the T3/junk leaks that a score threshold cannot (see leak analysis).

Head

• span feature: [start-word; end-word; mean-pool] from the frozen encoder word embeddings -> MLP (3·D -> 256 -> 256 -> 1), dropout 0.2
• training: 10 epochs, lr 0.001, AdamW, class-weighted BCE
• checkpoint selection: val AUROC

Holdout performance (eval_tiered_holdout, 15,332 docs)

metric	value
probe AUROC (all gold spans)	0.9482
baseline: GLiNER score AUROC (published)	0.869
probe AUROC (surface-matched subset)	0.9478
GLiNER score AUROC (same subset)	0.8837
surface-matched spans	20422

Threshold sweep

keep_recall: T1∪T2 spans retained. drop_expelled: T3/junk removed. residual_expelled: share of high-confidence leaks (GLiNER score

= 0.5, invisible to score thresholding) removed by the probe.

thr	keep_recall	drop_expelled	residual_expelled
0.30	0.9600	0.7355	0.2794
0.40	0.9463	0.7823	0.3947
0.50	0.9278	0.8133	0.4685
0.60	0.9076	0.8447	0.5473
0.70	0.8820	0.8736	0.6254

Probe score distributions

group	median (p25 / p75, n)
keep (T1∪T2)	0.986 (p25 0.914 / p75 0.998, n=16935)
drop (T3/junk)	0.019 (p25 0.000 / p75 0.329, n=4579)
tier3_nonmention	0.022 (p25 0.000 / p75 0.344, n=4322)
junk	0.002 (p25 0.000 / p75 0.061, n=257)
classifier_t3	-
residual leaks (GLiNER >= 0.5)	0.535 (p25 0.272 / p75 0.857, n=1396)

Artifacts

• head.pt — MLP state dict (input dim = 3 x encoder dim)
• probe_metrics.json — raw metrics (this page renders the summary)
• holdout_probe_predictions.jsonl — per-doc gold spans with probe_score and gliner_score per span

Reproduction: training/probe_gliner_tier.py in ai4data-playground.