Skip to content

Trusting your judges

An LLM judge is a model grading a model, so its scores are worth exactly as much as its agreement with a human. This page covers how AI Eval measures that agreement, the trust states it derives from the measurement, how to export the result for CI, and the configuration that decides which model judges and how it decodes. It is for whoever owns the eval numbers other people rely on.

Only LLM judges get a trust state and only they appear on the Judges tab. Deterministic graders are left out on purpose: their score is a fixed function of their code, so there is no human agreement to measure and no model drift to track. See graders for which is which.

Why a judge needs validating

A judge produces a number for every response, and a number looks like a measurement whether or not it is one. Validation is what turns it into one. You label a set of real responses yourself, the judge re-scores the same responses, and the two verdict sets are compared cell by cell.

Raw agreement hides the failure mode that matters. A judge that passes everything agrees with you on every response you passed, which reads as high accuracy while telling you nothing. So AI Eval reports two rates instead of one.

Rate Question it answers What a low value means
TPR, true positive rate Of the responses you passed, how many did the judge pass? The judge fails good work.
TNR, true negative rate Of the responses you failed, how many did the judge fail? The judge waves bad work through.

A judge is trusted when both rates reach 0.90, following Hamel Husain's eval guidance and Shreya Shankar's work on judge validation. The threshold is a constant in the report, not a per-site setting.

When your gold set contains only one class (all passes or all fails), the rate for the missing class is undefined. AI Eval stores it as unknown, never as 0.0, and every surface renders it as n/a. An undefined rate can never masquerade as a measured zero.

Building a gold set

Each judge has a gold labeling screen at /admin/config/ai/ai-eval/judges/{grader_id}/gold. It serves you one trace at a time, unlabeled ones first, and you mark each as pass, fail, or defer. Defer means "come back to this", so it does not count toward coverage.

Two scoping rules decide which traces you see and what your labels are worth:

  • Domain tags. A judge carries a list of domain tags, set on its Edit prompt tab. The tags define the judge's trace pool, so an accuracy judge for support answers is validated on support traces rather than on whatever happens to be imported. Saving tags never bumps the prompt version, so tagging a judge cannot invalidate a trusted validation of its current prompt.
  • Dimension. Every label written from the gold screen is stamped with the judge's dimension (accuracy, relevance, and so on). Validation reads only labels in the judge's own dimension, so an accuracy gold set never leaks into a tone judge's numbers. Labels written from the trace review queue carry no dimension: they are holistic labels, and an LLM judge does not read them.

A coverage counter tracks pass and fail labels across the judge's whole tag-scoped domain against the gold_target_n setting, 30 by default. The counter is computed over the full domain rather than the visible window, so it cannot overstate progress.

How reliable is the gold set itself

TPR and TNR compare the judge against a human, so neither can mean more than that human's labels do. If a labeler agrees with themselves 80% of the time on a dimension, a judge reported above that ceiling is measuring gold-set noise.

AI Eval measures the ceiling instead of assuming it. Span labels are stored per labeler and per round, so:

  • With two reviewers labeling the same traces, both verdicts persist and the screen reports Cohen's kappa alongside raw percent agreement and the overlap count.
  • With one reviewer, a blind re-label mode serves back traces you already labeled with your previous verdict hidden, stores the second pass as round 2, and reports your self-agreement the same way. Round 2 never feeds validation.

Undefined agreement (no overlap, or an overlap where only one class was used) is reported as undefined. The Judges tab shows the same reliability figure next to each trust state, so a TPR/TNR verdict is always read against the quality of the set behind it.

Running a validation

Open the judge from the Judges tab and use the launch form. It has four controls:

Control Values Effect
Gold labels All labels, Trace reviews only Which pool of human labels to draw from.
Hold-out split full, 70/30 full scores every available label. 70/30 scores a random 30% for a quicker run.
Threshold 0 to 5, default 3 The judge score at or above which a response counts as a pass, before it is compared to your label.
Split seed optional integer Only used with 70/30.

The run steps one example per poll and finishes with a confusion matrix, a disagreement inspector listing the false positives and false negatives, and a verbosity check reporting whether the judge favors long answers where your labels do not.

Seeds make a hold-out draw reproducible. Leave the field blank and a fresh seed is generated and recorded; enter a seed from an earlier run to reproduce that exact draw over the same gold-label pool. Adding or relabeling gold examples changes the pool, and with it the draw. A full run never draws, so it records no seed and any typed seed is ignored. Every validation is stored as a new attempt, so distinct seeds are visible evidence of distinct draws rather than a claim about them.

Sample size

The n behind a verdict is TP + FP + FN + TN. Examples the judge could not score are counted as skipped and listed separately, so they never inflate n. A verdict resting on 12 labels and one resting on 300 are both reported with their n, and the Judges tab also shows how many validation attempts have accumulated, so a judge validated repeatedly until it cleared the bar is distinguishable from one validated once.

Trust states

stateDiagram-v2
    [*] --> not_validated
    not_validated --> trusted: TPR and TNR both >= 0.90
    not_validated --> untrusted: either rate below 0.90
    trusted --> stale: report older than judge_trust_stale_days
    stale --> trusted: re-validated above the bar
    stale --> untrusted: re-validated below the bar
    untrusted --> trusted: re-validated above the bar

Trust decays with time. A trusted report older than judge_trust_stale_days (30 by default, and settable in configuration) reads as stale. The state is computed when it is read, so changing the window re-evaluates every stored report with no data migration. An untrusted verdict never decays into stale: it stays untrusted regardless of age.

Each report also records the prompt version it scored, so after you edit a judge's prompt the judge's page shows that the report was measured against the old wording.

Note

Trust is advisory. An untrusted judge never blocks a run and never changes an exit code. What it does change is presentation: every surface that shows a verdict the judge contributed to marks that verdict provisional and says why, because both gate metrics aggregate every grader. See running evaluations.

One thing validation cannot fix: a judge from the same model family as the model it grades is not an independent measurement. Models over-score their own family's output, so part of every score is the judge grading its own reflection, and the TNR your gold set measured no longer transfers cleanly. AI Eval compares the two families on every run and warns when they collide, though only for the openai, anthropic and google provider ids: under any other provider id every model reads as its own family and the collision goes undetected. The warning blocks nothing, since a same-family judge may be the only judge you have, but read the scores as family-inflated until a different family's judge is available.

The trust corpus, headless

The same corpus the Judges tab renders is readable from the CLI, one row per LLM judge. A judge that has never been validated is listed as not_validated rather than omitted, so counting your validated judges is a subtraction rather than a guess.

# Table: judge, trust, n, skipped, TPR, TNR, threshold, split,
# config version, seed, validated date, attempts
drush ai-eval:judges

# The same corpus as JSON, for CI and release checks
drush ai-eval:judges --json

ai-eval:judges (alias aej) is a reader and always exits 0.

The JSON is an array with one object per LLM judge and stable keys:

Key Type Notes
grader_id string The grader plugin id.
trust string trusted, untrusted, stale, or not_validated.
n integer TP + FP + FN + TN behind the latest verdict.
skipped integer Examples excluded from the matrix.
tpr, tnr float null when the rate is undefined.
threshold float The pass threshold that validation used.
hold_out_split string full or 70/30. Legacy null rows normalize to full.
config_version integer null when the shipped template was scored.
seed integer null when no hold-out draw was recorded.
validated_at integer Unix timestamp.
attempts integer Accumulated validation runs. 0 when never validated.

Every key except grader_id, trust, and attempts is null for a judge that has never been validated. A null is never collapsed to 0.

Validating from a fixture in CI

The UI path reads gold labels out of the database. The CLI path reads them from a YAML fixture, which is what a pipeline can commit and diff.

# Export a sample of stored results into a labeling fixture
drush ai-eval:sample-traces --grader=accuracy_grader --n=20 \
  --stratified=true --window-days=30 \
  --output=data/judge_validation/accuracy-labels.yaml

# After filling in human_label by hand, validate the judge against it
drush ai-eval:validate-judge --grader=accuracy_grader \
  --fixture=data/judge_validation/accuracy-labels.yaml

ai-eval:sample-traces (alias aest) takes --grader and --output (both required), --n (default 20), --stratified (default true, balances pass and fail rows where possible), and --window-days (default 30). It writes a meta block plus one row per example, with human_label left null for you to fill in.

ai-eval:validate-judge (alias aevj) takes:

Option Default What it does
--grader required The grader plugin id to validate.
--fixture required Path to the labeled YAML fixture.
--threshold 3.0 The binary pass boundary, on the shared 0 to 5 scale.
--json off Also print the report as JSON.
--force off Proceed when the fixture's meta.grader does not match --grader.
--resolution-confidence 0.9 Share of a gap's ranked pairs the judge must get right for that gap to count as resolved. Only used when the fixture declares a pairs block.

Exit codes make it a gate:

Code Meaning
0 The report is trusted.
3 Validation ran, and the judge is not trusted.
1 Bad input: missing options, unreadable or unparseable fixture, an invalid row, a malformed pairs block, or no labeled rows.

CLI runs are persisted like UI runs, so a fixture validation shows up in the judge's history. They score the full label set with no hold-out draw, so the row records no seed and no config version.

--threshold is expressed on the shared 0 to 5 scale and every grader score is converted to that scale before comparison, so one threshold means the same thing for every judge. A yes/no judge lands on exactly 0 or 5, so any threshold above 0 and up to 5 gives it the same verdict. If you hand-write a fixture whose grader_score values are in some other unit, declare the scale under meta:

meta:
  grader: groundedness_grader
  min_score: 0
  max_score: 1

Without it, the scale comes from the grader's own declaration, and failing that the scores are read as 0 to 5, which is what fixtures written before per-grader scales existed meant. Fixtures produced by ai-eval:sample-traces already carry the right meta.

Discrimination resolution

TPR and TNR say whether a judge is right about one answer. They do not say how far apart two answers must be before it can tell them apart, which is the question that matters when something claims a small improvement.

A fixture can answer it by declaring ranked pairs alongside its rows. Each pair names a better row, a worse row, and the true quality gap between them in the fixture's own score units:

pairs:
  - better: 12
    worse: 13
    gap: 1.0

The judge is credited with a pair only when it scores the better side strictly higher: a tie is not a ranking. Pairs are grouped by their declared gap, each gap is judged on its own pairs, and a gap counts as resolved when the one-sided 95% lower bound on its ranking rate clears --resolution-confidence. The reported resolution is the finest gap that holds up.

The bound is the gate, not the observed rate, so five correct rankings out of five establish nothing: the report publishes what the claim costs instead, which at the default confidence is 25 consecutive correct rankings at one gap. A judge that establishes nothing reports no figure rather than a zero, and pairs the judge scored on only one side are counted as unusable rather than dropped.

Resolution is measured on the fixture path only. The in-app validation screen works from your gold labels, which carry verdicts rather than ranked pairs, so it reports agreement rates and no resolution figure. The optimizer reads recorded resolutions to decide whether a candidate's gain is something any judge in the run can actually see.

The judge configuration surface

Decoding is pinned

The judge is the measurement instrument, so it should vary as little as the provider allows. Before every scoring call AI Eval pins the judge provider to temperature 0 and top_p 1.

That narrows judge variance. It does not remove it: no seed is pinned, and a hosted model can still return a different answer to an identical prompt. This is why the optimizer repeats every measurement rather than trusting a single run, and why the optimizer page describes judges as noisy. Read the pin as reducing avoidable variance, not as reproducibility.

Only knobs the model actually advertises are set, negotiated through the AI module's provider configuration. A provider that exposes neither knob gets no pin, and the step is a no-op rather than an error. Whether the pin applied is recorded in the run identity snapshot and exported with the envelope (judge_decoding_pinned), so "the judge was pinned" is a recorded fact about the run rather than an assumption about the code.

This is separate from the target's own sampling temperature. The target is the system under test and should normally run at its production temperature; see targets.

Which model judges

The judge provider and model are one site-wide pair, judge_provider and judge_model in ai_eval.settings, chosen on the settings form as a single provider and model option. Every LLM judge plugin resolves to that pair when it is instantiated. There is no per-judge model override: what a judge does carry per judge is its prompt template, its extra scoring guidance, and its domain tags.

Resolution order for one scoring call:

  1. The provider and model come from judge_provider and judge_model.
  2. The prompt template comes from the judge's stored override when one exists, otherwise from the template the plugin ships. Overrides are versioned, and the version scored is recorded on the validation report.
  3. The response handed to the judge is capped at judge_response_char_limit (4000 characters by default). This is a different cap from response_char_limit, which bounds the stored copy of a response.
  4. Decoding is pinned as described above.

Because the pair is site-wide and mutable, a run records the judge identity in force at the time. Result pages and exported envelopes read that snapshot rather than live settings, so changing the judge model tomorrow does not silently re-attribute yesterday's results. See sharing results.

References