Trusting your judges¶

An LLM judge is a model grading a model, so its scores are worth exactly as much as its agreement with a human. This page covers how AI Eval measures that agreement, the trust states it derives from the measurement, how to export the result for CI, and the configuration that decides which model judges and how it decodes. It is for whoever owns the eval numbers other people rely on.

Only LLM judges get a trust state and only they appear under Manage > Judges. Deterministic graders are left out on purpose: their score is a fixed function of their code, so there is no human agreement to measure and no model drift to track. See graders for which is which.

Why a judge needs validating¶

A judge produces a number for every response, and a number looks like a measurement whether or not it is one. Validation is what turns it into one. You label a set of real responses yourself, the judge re-scores the same responses, and the two verdict sets are compared cell by cell.

Raw agreement hides the failure mode that matters. A judge that passes everything agrees with you on every response you passed, which reads as high accuracy while telling you nothing. So AI Eval reports two rates instead of one.

Rate	Question it answers	What a low value means
TPR, true positive rate	Of the responses you passed, how many did the judge pass?	The judge fails good work.
TNR, true negative rate	Of the responses you failed, how many did the judge fail?	The judge waves bad work through.

A judge is trusted when both rates reach 0.90, following Hamel Husain's eval guidance and Shreya Shankar's work on judge validation. The threshold is a constant in the report, not a per-site setting.

When your gold set contains only one class (all passes or all fails), the rate for the missing class is undefined. AI Eval stores it as unknown, never as 0.0, and every surface renders it as n/a. An undefined rate can never masquerade as a measured zero.

Editing the prompt¶

The template must contain four %s placeholders (dimension, question, expectations, answer). Write %% for a percent sign. A template that does not fit is refused on save. A template saved earlier that no longer fits makes every answer it scores an error, with the reason on the result.

Answers the judge gave no score, because of an error or because it did not apply, are shown as Not scored on the report, history and Judges page. They are left out of the matrix, rates and repeat diagnostics. Any such answer blocks trust; check the judge and validate again. The judge comparison leaves out and counts answers a judge gave no score on any try.

Building a gold set¶

The gold-label and blind re-label screens need validate ai eval judges, which also permits saving trace labels for judge validation. Promoting a trace into a dataset additionally needs annotate ai eval results or administer ai eval. The promotion control is hidden when you lack that permission. Importing traces needs operate ai eval or administer ai eval.

Each LLM judge that scores with its own prompt has a gold labeling screen at /admin/config/ai/ai-eval/judges/{grader_id}/gold, opened from its Label traces tab. The decision rubric judge and content review judges have neither the tab nor the page. The screen serves one trace at a time in a stable order. Mark it pass, fail, or defer. Saving advances to the next unlabeled trace, wrapping to the first one when needed. Deferred traces keep their place and are skipped during this advance. Next unlabeled returns to the first trace you have not labeled in this pass. The last trace stays on screen and shows Pass complete when the pass is done. Defer means "come back to this", so it does not count toward coverage.

Only final answers enter the gold pool: the normalized reply must have non-empty text and no response tool calls. A provider's finish reason does not decide this. The screen filters before filling its 200-answer window, so tool steps do not crowd answers out. Coverage and agreement use the same rule over the full domain. A domain containing only tool steps or empty replies explains that it has no final answers yet.

Labels already stored on non-final spans are kept, but are left out of validation. The judge page reports how many such labels were not used.

Two scoping rules decide which traces you see and what your labels are worth:

Domain tags. A judge carries a list of domain tags, set on its Edit prompt tab. The tags define the judge's trace pool, so an accuracy judge for support answers is validated on support traces rather than on whatever happens to be imported. Saving tags never bumps the prompt version, so tagging a judge cannot invalidate a trusted validation of its current prompt.
Dimension. Every label written from the gold screen is stamped with the judge's dimension (accuracy, relevance, and so on). Validation reads only labels in the judge's own dimension, so an accuracy gold set never leaks into a tone judge's numbers. Labels written from the trace review queue carry no dimension: they are holistic labels, and an LLM judge does not read them.
Results page reviews. The pass or fail people give an answer on a results page counts only for a judge without a dimension (the decision rubric judge, a content review judge), and only when that judge scored that answer in that run: it gave a score, zero included, without an error. A review of an answer from a test the judge was not on, or that the judge failed to score or did not apply to, says nothing about it and is left out. A review that counts is still a verdict on the whole answer while the judge checks only its own rules, so the validation report and the calibration page say so: an answer failed for a wrong fact counts against a judge that checks only style.
Reviews from before the upgrade. Results stored before AI Eval recorded which judges scored each answer cannot say which judge a review is about, so their reviews no longer count for any judge's validation. They are kept, not deleted, and nothing guesses their judges from today's tests. The judge's validation page says how many were left out; run the test again and review the new results to validate on them. A judge with domain tags never reads results-page reviews (they carry no tags), so its page does not count them.

A coverage counter tracks pass and fail labels across the judge's whole tag-scoped domain against the gold_target_n setting, 30 by default. The counter is computed over the full domain rather than the visible window, so it cannot overstate progress.

The blind re-label pass keeps the order of your first-pass label ids. Saving a verdict advances one position without revealing your earlier verdicts.

The Deferred: N count tracks deferred answers in the review window. Once every answer there is labeled or deferred, the page says All answers are labeled or deferred. N deferred. Use Review deferred answers to return to the first deferred answer. Defer never counts toward gold coverage.

Continuing an unfinished validation¶

Return to the judge page to continue its newest unfinished validation. The page shows how many answers have been scored and resumes polling. It uses the labels saved when the run started, even if you have since changed them. The launch form stays hidden while that run is open. Use Cancel to stop it and restore the launch form. Leaving the page pauses progress until you return; scoring still happens through the page's protected POST requests.

How reliable is the gold set itself¶

TPR and TNR compare the judge against a human, so neither can mean more than that human's labels do. If a labeler agrees with themselves 80% of the time on a dimension, a judge reported above that ceiling is measuring gold-set noise.

AI Eval measures the ceiling instead of assuming it. Span labels are stored per labeler and per round, so:

With two reviewers labeling the same traces, both verdicts persist and the screen reports Cohen's kappa alongside raw percent agreement and the overlap count.
With one reviewer, a blind re-label mode serves back traces you already labeled with your previous verdict hidden, stores the second pass as round 2, and reports your self-agreement the same way. Round 2 never feeds validation.

Undefined agreement (no overlap, or an overlap where only one class was used) is reported as undefined. Manage > Judges shows the same reliability figure next to each trust state, so a TPR/TNR verdict is always read against the quality of the set behind it.

Running a validation¶

A judge this site cannot run, such as the decision rubric judge when the AI module has no Decision operation, reads Not available on Manage > Judges. Its page says why and offers no validation, and the test forms show it disabled with the same reason. A test saved earlier with that judge keeps it.

Open the judge from Manage > Judges and use the launch form. It has four controls:

Control	Values	Effect
Gold labels	All labels, Trace reviews only	Which pool of human labels to draw from.
Hold-out split	`full`, `70/30`	`full` scores every available label. `70/30` scores a random 30% for a quicker run.
Threshold	0 to 5	The judge score at or above which a response counts as a pass, before it is compared to your label. Starts at the latest validation's threshold; with none, the field is empty and required.
Split seed	optional integer	Only used with `70/30`.

The threshold is on the 0 to 5 scale every judge is compared on, whatever the judge's own scale: validation normalizes each score first. The form says what that means for the judge: on a judge that scores 0 to 100, 4.5 is 90, and on a judge that scores 0 to 1, 2.5 is 0.5. The form also lists the tests that use the judge and that you can see, with their question pass threshold, and marks one that differs from the latest validation. A test's question threshold applies to the average of all its graders, not to one judge, so it is context, not a value to copy. The validation history and the run comparison show the threshold each validation ran at.

The run steps one example per poll and finishes with a confusion matrix, a disagreement inspector listing the false positives and false negatives, and a verbosity check reporting whether the judge favors long answers where your labels do not. The matrix reads "Human: pass / fail" against "Judge: pass / fail" whatever the judge's dimension, and its verdict uses the same words as the trust badges: Trusted or Failed validation. For each disagreement the inspector shows the judged response, your verdict and note, and the judge's verdict and its own reason; long text folds into an expandable block.

Seeds make a hold-out draw reproducible. Leave the field blank and a fresh seed is generated and recorded; enter a seed from an earlier run to reproduce that exact draw over the same gold-label pool. Adding or relabeling gold examples changes the pool, and with it the draw. A full run never draws, so it records no seed and any typed seed is ignored. Every validation is stored as a new attempt, so distinct seeds are visible evidence of distinct draws rather than a claim about them.

Sample size¶

The n behind a verdict is TP + FP + FN + TN. Answers the judge gave no score are counted as not scored, apart from skipped labels, so they never inflate n. Any not-scored answer blocks trust: check the judge and validate again. A verdict resting on 12 labels and one resting on 300 are both reported with their n, and Manage > Judges also shows how many validation attempts have accumulated, so a judge validated repeatedly until it cleared the bar is distinguishable from one validated once.

A judge is trusted only after at least 10 distinct pass labels and 10 distinct fail labels were scored, with both rates at least 0.90. Repeated scores of one answer count once. Below that floor, Not enough labels is a warning to add labels and validate again. The gold_target_n setting is a labeling target, not this minimum. Full uses every label; a 70/30 split of a small set rarely reaches the minimum.

Comparing two validations¶

The judge's Compare runs tab sets two of its validations side by side: TPR and TNR, the length check, the smallest gap the judge can tell apart, the threshold, the verdict, the config version and the seed, with the change from the older to the newer. It starts on the newest two; pick any two of the judge's last 50 validations to compare those instead. The comparison has its own address, so you can bookmark or share it. When the two found exactly the same figures, which is common for repeats of one run, the page says so above the table.

Trust states¶

stateDiagram-v2
    [*] --> not_validated
    not_validated --> not_enough_labels: fewer than 10 pass or fail labels scored
    not_enough_labels --> trusted: re-validated on enough labels
    not_validated --> trusted: TPR and TNR both >= 0.90
    not_validated --> untrusted: either rate below 0.90
    trusted --> stale: report older than judge_trust_stale_days
    trusted --> stale: judge now uses a different model
    trusted --> stale: prompt edited after the report
    stale --> trusted: re-validated above the bar
    stale --> untrusted: re-validated below the bar
    untrusted --> trusted: re-validated above the bar

Pages and the CLI name the states the same way: Trusted, Not validated (not_validated), Not enough labels (not_enough_labels), Failed validation (untrusted) and Validation out of date (stale). A test with no LLM judge, only graders such as Format, reads No judge: there is nothing to validate. The dashboard's Judge not trusted count covers every test whose worst judge is in one of the four states that are not trusted, never-validated judges included, and leaves out tests with no judge.

Trust decays with time. A trusted report older than judge_trust_stale_days (30 by default, and settable in configuration) reads as stale. The state is computed when it is read, so changing the window re-evaluates every stored report with no data migration. An untrusted verdict never decays into stale: it stays untrusted regardless of age.

A trusted report also reads as stale when the judge now uses a different provider or model than the one recorded on the report, because the verdict measured a judge that no longer runs. Stale, not untrusted: nothing showed the new model to be wrong, it has simply not been measured.

A trusted report also reads as stale when you edit the judge's template or guidance after the report. The report names the version it measured; the Judges page shows it as v1 (now v2). A report on the default prompt reads as stale once any override is saved. Saving only the model or domain tags does not change the prompt version; a changed model still makes it stale by the model rule. Fixture validations use the scores in the file and measure no prompt, so they read as stale once the judge has an override. Validate in the browser to measure the override. Trust is read when a page opens, so a result stored earlier also loses its Trusted mark and corrected rate after a prompt edit.

Note

Trust is advisory by default. A judge that is not trusted still scores, and every verdict it contributed to is marked provisional with the reason. With refuse_unvalidated_judges on (Do not count a pass that rests on a judge that is not trusted in Settings), such a gate pass is recorded as a fail. A hard gate then makes Drush exit 1; a soft gate does not. The setting applies to runs that finish after the change; earlier results keep their verdict. See running evaluations.

One thing validation cannot fix: a judge from the same model family as the model it grades is not an independent measurement. Models over-score their own family's output, so part of every score is the judge grading its own reflection, and the TNR your gold set measured no longer transfers cleanly. AI Eval compares the two families on every run and warns when they collide. Vendor-namespaced aggregator model ids such as anthropic/claude-sonnet-4 resolve to the underlying family; identities the heuristic cannot resolve are shown as unknown rather than as a verified different family. The warning blocks nothing, since a same-family judge may be the only judge you have, but read the scores as family-inflated until a different family's judge is available.

The trust corpus, headless¶

The same corpus rendered under Manage > Judges is readable from the CLI, one row per LLM judge. A judge that has never been validated is listed as not_validated rather than omitted, so counting your validated judges is a subtraction rather than a guess.

# Table: judge, trust, n, skipped, not scored, TPR, TNR, threshold, split,
# config version, seed, validated date, attempts
drush ai-eval:judges

# The same corpus as JSON, for CI and release checks
drush ai-eval:judges --json

ai-eval:judges (alias aej) is a reader and always exits 0.

The JSON is an array with one object per LLM judge and stable keys:

Key	Type	Notes
`grader_id`	string	The grader plugin id.
`trust`	string	`trusted`, `untrusted`, `not_enough_labels`, `stale`, or `not_validated`.
`n`	integer	TP + FP + FN + TN behind the latest verdict.
`skipped`	integer	Examples excluded from the matrix.
`not_scored`	integer	Labeled answers the judge gave no score; `null` before this was recorded.
`tpr`, `tnr`	float	`null` when the rate is undefined.
`threshold`	float	The pass threshold that validation used.
`hold_out_split`	string	`full` or `70/30`. Legacy `null` rows normalize to `full`.
`config_version`	integer	`null` when the shipped template was scored.
`seed`	integer	`null` when no hold-out draw was recorded.
`validated_at`	integer	Unix timestamp.
`attempts`	integer	Accumulated validation runs. `0` when never validated.

Every key except grader_id, trust, and attempts is null for a judge that has never been validated. A null is never collapsed to 0.

Validating from a fixture in CI¶

The UI path reads gold labels out of the database. The CLI path reads them from a YAML fixture, which is what a pipeline can commit and diff.

Both paths add a row to the judge's validation history, and its Labels column says which pool a row used: this site for a validation run in the browser, file: followed by what the file says about its labels for one given to drush, and not recorded for rows from before the column existed. For ai-eval:validate-judge that is the fixture's meta.labels, else meta.label_source, else the file's name. For the content review module's ai-eval-cr:import-run it is meta.label_source, else meta.labels, else a note that the fixture does not say where its labels came from. Run validation in the browser never reads a file, so a judge validated only from files has no labels for it: the page then says so and shows the drush command for that judge.

# Export a sample of stored results into a labeling fixture
drush ai-eval:sample-traces --grader=accuracy_grader --n=20 \
  --stratified=true --window-days=30 \
  --output=data/judge_validation/accuracy-labels.yaml

# After filling in human_label by hand, validate the judge against it
drush ai-eval:validate-judge --grader=accuracy_grader \
  --fixture=data/judge_validation/accuracy-labels.yaml

ai-eval:sample-traces (alias aest) takes --grader and --output (both required), --n (default 20), --stratified (default true, balances pass and fail rows where possible), and --window-days (default 30). It writes a meta block plus one row per example, with human_label left null for you to fill in.

ai-eval:validate-judge (alias aevj) takes:

Option	Default	What it does
`--grader`	required	The grader plugin id to validate.
`--fixture`	required	Read human labels from this YAML fixture path.
`--threshold`	`3.0`	The binary pass boundary, on the shared 0 to 5 scale.
`--json`	off	Also print the report as JSON.
`--force`	off	Proceed when the fixture's `meta.grader` does not match `--grader`.
`--resolution-confidence`	`0.9`	Share of a gap's ranked pairs the judge must get right for that gap to count as resolved. Only used when the fixture declares a `pairs` block.

Exit codes make it a gate:

Code	Meaning
`0`	The report is trusted.
`3`	Validation ran, and the judge is not trusted.
`1`	Bad input: missing options, unreadable or unparseable fixture, an invalid row, a malformed `pairs` block, or no labeled rows.

A fixture needs at least 10 distinct scored pass and 10 distinct scored fail items; otherwise the command exits 3.

CLI runs are persisted like UI runs, so a fixture validation shows up in the judge's history. They score the full label set with no hold-out draw, so the row records no seed and no config version. It measures no prompt, so it reads as out of date once the judge has a prompt override; validate in the browser to measure that override.

--threshold is expressed on the shared 0 to 5 scale and every grader score is converted to that scale before comparison, so one threshold means the same thing for every judge. A yes/no judge lands on exactly 0 or 5, so any threshold above 0 and up to 5 gives it the same verdict. If you hand-write a fixture whose grader_score values are in some other unit, declare the scale under meta:

meta:
  grader: groundedness_grader
  min_score: 0
  max_score: 1

Without it, the scale comes from the grader's own declaration, and failing that the scores are read as 0 to 5, which is what fixtures written before per-grader scales existed meant. Fixtures produced by ai-eval:sample-traces already carry the right meta.

Discrimination resolution¶

TPR and TNR say whether a judge is right about one answer. They do not say how far apart two answers must be before it can tell them apart, which is the question that matters when something claims a small improvement.

A fixture can answer it by declaring ranked pairs alongside its rows. Each pair names a better row, a worse row, and the true quality gap between them in the fixture's own score units:

pairs:
  - better: 12
    worse: 13
    gap: 1.0

The judge is credited with a pair only when it scores the better side strictly higher: a tie is not a ranking. Pairs are grouped by their declared gap, each gap is judged on its own pairs, and a gap counts as resolved when the one-sided 95% lower bound on its ranking rate clears --resolution-confidence. The reported resolution is the finest gap that holds up.

The bound is the gate, not the observed rate, so five correct rankings out of five establish nothing: the report publishes what the claim costs instead, which at the default confidence is 25 consecutive correct rankings at one gap. A judge that establishes nothing reports no figure rather than a zero, and pairs the judge scored on only one side are counted as unusable rather than dropped.

Resolution is measured on the fixture path only. The in-app validation screen works from your gold labels, which carry verdicts rather than ranked pairs, so it reports agreement rates and no resolution figure. The optimizer reads recorded resolutions to decide whether a candidate's gain is something any judge in the run can actually see.

The judge configuration surface¶

Decoding is pinned¶

The judge is the measurement instrument, so it should vary as little as the provider allows. Before every scoring call AI Eval pins the judge provider to temperature 0 and top_p 1.

That narrows judge variance. It does not remove it: no seed is pinned, and a hosted model can still return a different answer to an identical prompt. This is why the optimizer repeats every measurement rather than trusting a single run, and why the optimizer page describes judges as noisy. Read the pin as reducing avoidable variance, not as reproducibility.

Only knobs the model actually advertises are set, negotiated through the AI module's provider configuration. A provider that exposes neither knob gets no pin, and the step is a no-op rather than an error. Whether the pin applied is recorded in the run identity snapshot and exported with the envelope (judge_decoding_pinned), so "the judge was pinned" is a recorded fact about the run rather than an assumption about the code.

Some models refuse the knobs their provider advertises. Claude Haiku 5.5, for example, answers a request with temperature set by an error. When the error names a pinned knob and says it is not accepted, the judge asks again without the pin, and stays unpinned for the rest of the run, so the model costs one extra call per run rather than one per question. The run identity then records the judge as not pinned. Any other error still fails the judge as before.

To use the provider's own decoding instead, turn off Pin judges to temperature 0 on the AI Eval settings form (pin_judge_decoding in ai_eval.settings). It is on by default.

This is separate from the target's own sampling temperature. The target is the system under test and should normally run at its production temperature; see targets.

Which model judges¶

Every LLM judge uses the site judge unless it sets its own model. The site judge is one provider and model pair, judge_provider and judge_model in ai_eval.settings, chosen on the settings form. A judge's own model is set on its Edit prompt tab, in the Judge model section, from the same provider and model list. Leave it on Use the site judge model to follow the site judge. The override is stored with the judge's other per-judge settings in ai_eval.judge_config.{grader_id} as judge_provider and judge_model, and it applies only when both are set.

Changing a judge's model is not a prompt edit, so it never creates a new prompt version. It does change what the judge measures, so it is tracked the other way round: every validation records the provider and model that judged it, and a trusted verdict earned on a different model than the one the judge uses now reads as stale until you validate again. Manage > Judges shows each judge's current model (marked "own model" or "site judge") and, when they differ, the model its latest validation used. Validations recorded before per-judge models existed carry no model, show "not recorded", and keep their verdict.

Resolution order for one scoring call:

The provider and model come from the judge's own override when both are set, otherwise from judge_provider and judge_model.
The prompt template comes from the judge's stored override when one exists, otherwise from the template the plugin ships. Overrides are versioned, and the version scored is recorded on the validation report.
The response handed to the judge is capped at judge_response_char_limit (4000 characters by default). This is a different cap from response_char_limit, which bounds the stored copy of a response.
Decoding is pinned as described above.

The judge's reason is kept up to 1000 characters, and the disagreement inspector shows it next to the human reviewer's own note and the judged response.

Because both the site judge and the per-judge models are mutable, a run records the judge identity in force at the time. The run snapshot carries judge_models, one entry per LLM grader with the provider, model, and whether it came from the judge's own setting or the site judge. judge_provider and judge_model keep naming the run's judge: the shared pair when all of the run's judges use one model, and mixed when they differ. Result pages and exported envelopes read that snapshot rather than live settings, so changing a judge's model tomorrow does not silently re-attribute yesterday's results. See sharing results.

The same-family warning on the dashboard compares each LLM grader's own model with the target's model, so a judge that has its own model is checked on that model rather than on the site judge.

References¶

Hamel Husain, Evals FAQ
Shreya Shankar et al., Who Validates the Validators?