Claude Sonnet 4.6 invented a tiebreaker rule on data that could not be reconciled, 3 runs of 3: two systems of record, identical timestamps, equal authority, and the only correct answer was to stop. The run, with the model's answer verbatim, is in Tessera, First Contact. This phile is the reasoning behind the eval.
What a leaderboard does not measure
A public benchmark scores the answer it was handed. Its questions were curated so that each has a clean, knowable truth, which leaves no case where refusing is correct. A top score on that dataset measures capability, not whether you can trust the answer.
Your own corpus is not curated. Two systems of record disagree about the same fact. Context goes stale and carries no label. Some answers were never written down. A number for that has to be built on your own data.
Three axes, and where refusal lives
I grade three things. Accuracy: is the answer right. Provenance: does it cite the source that backs the claim. Correct refusal: does it decline when the data is missing or the sources conflict.
Accuracy is a scalar you already know how to climb. Refusal decides whether a confidently wrong answer reaches a customer, and a pure accuracy metric penalizes it. The model writes code and prose well and keeps improving without my help. The part I measure is reasoning over scattered, contradictory company knowledge and stopping when it should.
A model cannot grade itself
The obvious way to score refusal is to ask the model whether it was sure enough to answer. The score a model gives itself follows its confidence, and the two separate exactly when the model cannot find the answer: it invents one and rates the invention highly.
So the verdict comes from outside the model. In my harness the gates are deterministic: a bash assertion compares the cited source with the one that should have been cited, whatever the explanation sounds like. In Tessera the grader is a model from a different lab (GPT-4o grading Claude), and the eval aborts if the grader resolves to the model under test.
You measure it more than once
A green demo is one sample. The agent that refused correctly this morning can cite a source that does not exist this afternoon, on the same question with the same prompt. I measure pass^k: run the question k times and count how often all three axes hold together. A probe that passes once and fails twice averages 33% and scores 0 on pass^3. On the Sonnet run the score was pass^3 75%, and the missing 25% is the probe above.
Tessera is an open methodology and generator for reliability evals over MCP, built on Inspect: github.com/rinaldofesta/tessera.
Where the metric is weak
Deciding what counts as a correct refusal is itself a judgment call, and I still do not have a clean rule for it. An agent that refuses everything scores a perfect refusal rate and is useless, so the metric only means something once you also penalize the answers it held back that it should have given. Drawing that line is harder than three tidy axes make it sound, and I keep getting it slightly wrong. The reference organization is synthetic, with 4 probes, and I fixed 3 bugs in my own harness before the first numbers meant anything.