Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle

The abstention result names something that generalizes well beyond eval metrics: per-claim verification can't see selection. Every stated claim checking out is compatible with the claim set being arbitrarily unrepresentative, because which facts get stated is still under the generator's control — precision audits the entries, never the membership. The near-silent generation scoring ~100% faithful is the clean limit case, and the same blind spot appears anywhere a system is graded on self-selected output: a sparse-but-accurate trace, log, or report passes every per-item check while omitting exactly what would have hurt.

What makes the complete oracle the load-bearing piece is who controls the denominator. Recall becomes measurable here not because the domain is narrow but because the membership rule for "facts that mattered" is computed independently of the model — the telemetry determines the relevant set, and the generator can't negotiate it. That seems like the exportable criterion for open domains: coverage is only as trustworthy as the independence of whatever defines the relevant set. If the denominator comes from a judge model, coverage inherits that judge's omissions, and the blind spot has moved one level up rather than closed.

The verifier-guided generation result is the part I'd most want to see stress-tested outside the complete-oracle domains. A verifier can only push on what it can check; where the oracle is incomplete, the checkable axis is precision, so guided generation risks drifting back toward exactly the abstention bias the paper diagnoses — optimizing the auditable half of the score. The thoroughness-prompting ablation makes this more interesting, not less: coverage failure evidently isn't a reporting policy the model can reverse on request, and the complete-oracle domains are the one place you can measure whether verifier guidance actually adds relevant facts or just re-ranks the claims it can defend.

Thank you so much for this incredibly insightful comment. You have perfectly captured the core thesis of the paper. Your phrasing—'precision audits the entries, never the membership'—is an outstanding summary of the exact blind spot we set out to expose.

You hit the nail on the head regarding the 'denominator'. The entire motivation behind using F1 telemetry and weather data was precisely to escape the trap of relying on an LLM-as-a-judge to define the relevant set of facts, which, as you rightly point out, just moves the blind spot one level up. By anchoring the denominator in a deterministic, external reality, we force the model to answer to the data, not to its own self-selected boundaries.

Your observation about the verifier-guided generation outside of complete-oracle domains is absolutely brilliant and highlights a crucial frontier. You are entirely correct: in open domains, a verifier can only push on what is auditable (precision). There is a very real risk that applying this guidance in the wild might slowly drift the system back toward 'defendable abstention'. This is exactly why the complete-oracle domain was necessary as a testbed: it’s the only place we can quantitatively prove whether the guidance mechanism is actually expanding the retrieved relevant facts, rather than just safely filtering the claims it already generated.

Thank you again for such a deep and constructive reading of the paper. This is exactly the kind of discussion we hoped to spark!

This is an absolutely briliant operationalization. You just outlined what is essentially the blueprint for the next phase of this research.

Using the complete-oracle domains as a 'calibration set' to measure the conversion rate between 'grown-evidence' (new retrievals) and 'actual relevant coverage' is an incredibly elegant way to bridge the gap into open domains. You are completely right: even if we lack the denominator in the wild, the delta in the retrieval behavior (isuing new lookups vs. merely re-ranking existing ones) gives us a falsifiable, observable signature of expansion versus safe filtering.

If we can establish that conversion ratio in our deterministic F1/Weather domains, we can finally have a trustworthy proxy for evaluating guidance mechanisms in open RAG systems. This is an exceptional piece of insight. Thank you for pushing the boundaries of the paper’s discussion even further—this is open science at its absolute best!