Scoring and quality gates¶

How per-grader scores become a question score, how question scores become a target verdict, how the quality gate is applied, and what ends up stored. For operators reading a result and for anyone wiring a gate into CI.

The three levels¶

flowchart TD
    G[Grader scores, each on its own declared scale] --> C[Mapped onto the composite 0 to 5 scale]
    C --> Q[Question avg_score, the mean of the mapped scores]
    Q --> P{avg_score >= question pass threshold?}
    P -->|yes| MP{Any must-pass grader below the threshold?}
    MP -->|no| QP[Question passed]
    MP -->|yes| QF
    P -->|no| QF[Question failed]
    Q --> T[Target avg_score, the mean over measured questions]
    QP --> R[Target pass_rate, the share that passed]
    QF --> R
    T --> GATE{Gate metric}
    R --> GATE
    GATE --> V[passed_gate]

Question score¶

A question's avg_score is the plain mean of its grader scores, rounded to two decimal places. Three rules decide which scores enter that mean.

Scores are mapped onto a shared scale first. Graders declare their own range. Averaging a binary judge's 1 with a gradient judge's 4 as raw numbers would be meaningless, so every score is converted onto the composite 0 to 5 scale using that grader's declared minimum and maximum. A binary yes becomes 5, a binary no becomes 0. A grader that already declares 0 to 5 goes through an explicit identity path and its number is not touched at all, so results from before per-grader scales existed reproduce exactly. See graders for which grader uses which scale.

Null scores are excluded, not counted as zero. A grader that does not apply to the question (no expected_facts, no expected_tools, no executable rubric checks) contributes nothing to the mean.

Errored graders are excluded and flag the question. A grader that could not run is a different matter from one that did not apply. It is excluded from the mean and it sets an error key on the question reading grader error: <grader ids>. That flag is what keeps the question out of the gate later, so a PASS built only on the deterministic graders while every judge was unreachable cannot happen.

A question with no valid scores is marked unscored. It remains distinct from both a genuine zero and a grader error, is excluded from the target average and pass-rate denominator, and is reported in the run's unscored count.

This is reachable with an ordinary configuration. Attach only fact_match_grader and tool_usage_grader, then include a question that carries neither expected_facts nor expected_tools, and it is marked unscored because neither grader applies. Check your grader list against the shape of your dataset.

The question passes when it is scored, has no errored grader, and its avg_score meets the question pass threshold, and no must-pass grader scored below that threshold. The threshold resolves in this order: the target's own question_pass_threshold, then the site-wide question_pass_threshold setting, then 3.5.

There are no per-grader weights¶

Every grader that produced a usable score counts equally. Attaching four judges and one deterministic check gives the judges four fifths of the question score, and the only way to change the balance is to change the grader list.

Weights do exist one level down, inside a rubric: a rubric's scoring block can combine its checks with weighted_avg and a per-check weight map, and the combined verdict then arrives as the single rubric_checks grader score. See rubrics.

Must-pass graders¶

Mark a selected grader as Must pass in the test editor or guided step 1. Its own score, mapped onto the composite 0 to 5 scale and rounded to two places, must reach the same question pass threshold as the average. Equality passes. With scores 5, 5, 5, 0, the average is 3.75; requiring the last grader makes the question fail even though its average passes the default 3.5.

A required grader that does not apply contributes no failure. An errored grader still excludes the question from target averages. If a different required grader measured a failure on that question, that failure still fails the gate; the result row shows both reasons. Unselected grader ids have no effect.

Target score¶

Questions carrying an error key or an unscored marker are dropped before aggregation. Legitimate zeros are kept. A question that genuinely scored 0.0 is a measurement and belongs in the average; a question whose graders could not run is an absence of measurement and does not.

From the surviving questions:

Figure	How it is computed
`avg_score`	Mean of the surviving question `avg_score` values, rounded to 2 dp
`pass_rate`	Share of surviving questions that passed, rounded to 2 dp
`pass_rate_ci_low` / `pass_rate_ci_high`	Wilson score interval at 95% (z = 1.959964) over the same counts
`valid_count`	How many questions survived

The Wilson interval is stored and displayed alongside every pass rate, so a rate from twelve questions visibly carries more uncertainty than one from four hundred.

The coverage guard¶

If fewer than half of a target's questions could be scored, the surviving sample is too small to trust and the gate fails regardless of what the survivors scored. A total outage, where nothing survived, is the degenerate case of the same rule and produces a row of zeros with passed_gate false.

Coverage, must-pass graders, and the judge trust refusal setting can each force a gate verdict to fail. A transient judge outage must not read as a PASS on a lucky subset.

The corrected pass rate¶

When a test has exactly one grader and it is a trusted LLM judge, the result page also shows a pass rate corrected for the judge's measured agreement with your labels (TPR and TNR). The correction requires validation at the run's question pass threshold. When they differ, the result shows Not available and both thresholds. Validate the judge at the threshold the run used. Results stored without a recorded threshold show Not available too.

A validation whose scored answers were all at the scale endpoints (0 or 5) can be reused at any threshold above 0 and up to 5. This is measured from the scores; a judge's name or declared range does not establish it. Validations from before this release need to be run again to record that evidence. The corrected rate does not change the gate. Pass thresholds accept at most two decimals; validation and run scoring both round scores to two decimals before deciding whether they pass.

The quality gate¶

A gate is a metric plus a threshold, and the two live on different scales. The comparison is always measured figure >= threshold.

Metric	Scale	Default threshold	Displayed as
`avg_score`	0 to 5	4.0	`4.0`
`pass_rate`	0 to 1	0.8	`80%`

The same number means opposite things on the two scales. 0.8 is a sensible pass_rate gate and a near-guaranteed pass as an avg_score gate; 4.0 is a sensible avg_score gate and unreachable as a pass_rate gate.

Validation is metric-aware. The widget is not. The threshold field uses a single step of 0.05 whatever the metric, because it does not re-render when you change the metric and 0.05 divides both scales. What does look at the metric is validation on save: it blocks a gate that could never pass, a pass_rate above 1.0 or an avg_score above 5, and it warns without blocking when you set an avg_score threshold of 1.0 or less, because that passes almost any run and usually means you wanted pass_rate. There is no equivalent warning for an implausibly small pass_rate.

Any stored metric value that is not exactly pass_rate resolves to avg_score, including an empty or unrecognized one. That keeps an old or hand-edited config readable rather than fatal.

Since every grader is mapped onto the composite scale before averaging, the avg_score ceiling really is 5. A grader declaring a wider range no longer lifts it, so an avg_score threshold above 5 is unreachable by any grader selection.

Must-pass failures¶

Any measured must-pass failure fails the gate under either metric, even when the average score or pass rate clears its threshold. A soft gate still only warns. The command and result page name the question and grader.

Average scores and aggregation formulas stay unchanged. Pass rate and its Wilson interval do change when a must-pass failure changes a measured question's verdict: one failure among four previously passing questions lowers the pass rate from 1.00 to 0.75.

Hard and soft gates¶

A target's gate field is hard or soft, defaulting to hard.

drush ai-eval:run exits 1 when any target with a hard gate failed its gate, and prints One or more hard gates failed.. A soft gate is reported the same way in the output but does not affect the exit code. Nothing else in the module changes behavior based on hard versus soft. See running for the CI wiring.

Warnings and judge trust refusal¶

The judge and target sharing a model family and a dataset below the sample-size floor produce warnings without changing the verdict or exit code. Judge trust is also advisory by default. With refuse_unvalidated_judges on (Do not count a pass that rests on a judge that is not trusted in Settings), a gate pass that rests on a judge that is not trusted is recorded as a fail. A hard gate then makes Drush exit 1; a soft gate does not. Scores stay as measured, and earlier results keep their verdict. See judges for trust states.

Runs are gated against the recorded threshold¶

A stepped run started from the UI is scored against the metric and threshold captured in its run-identity snapshot at start time, not against the live target config. Editing a target's gate halfway through a run does not decide the verdict of the run that began under the old one. A run started before snapshots existed falls back to live config for older settings, but always uses an empty must-pass list. A snapshot without must_pass also means no required graders, so later target edits cannot add them to an old run.

What is persisted¶

One row per completed target run in ai_eval_result:

Column	Contents
`target_id`, `timestamp`	Which target, and when the row was written
`avg_score`, `pass_rate`	The target-level figures above
`pass_rate_ci_low`, `pass_rate_ci_high`	The Wilson bounds
`passed_gate`	The gate verdict, as 0 or 1
`question_count`	Every question attempted, including errored ones
`error_count`	How many questions carry an `error` key
`scores_json`	The full per-question detail, described below
`source`	How the run was triggered (`manual`, `cron`, `ui`)
`duration`	Seconds the run spent working. A run from the UI that was interrupted and resumed leaves the pause out. UI runs recorded before active-time tracking stored the wall-clock span instead, which the result page labels as wall clock
`run_id`	The stepped-run row that produced it, when run from the UI
`config_json`	The run-identity snapshot: provider, model, prompt, dataset and rubric hashes, judge config, retries, and the start-time `must_pass` list

scores_json holds one entry per question with:

question_id, input, avg_score, passed, and error when the question was flagged;
must_pass_failed, a list of grader ids, only when non-empty;
grader_scores, keyed by grader ID, each carrying score, reason, grader_id, error, min_score and max_score. The score is the grader's raw answer, not the mapped one, and the recorded min_score and max_score are what make a stored 1 from a binary judge readable as a full pass without looking up a plugin definition that may have changed since. A grader that answers with a decision model's pass probability (the decision rubric judge) also stores decision: the raw p_pass, the fingerprint, provider and model it came from, the rubric question, and the pass_score and points_per_error it used. Calibrations are fitted from it (calibrating a decision judge);
elapsed, and raw_response capped at the target's response character limit or 8000 characters, whichever is smaller;
token_usage (with cached when the provider reports it), model_id and model_id_resolved in chat mode and agent mode, where the provider reports them. Agent mode records the chat calls the agent makes, retries included; the totals are left out when any call reports no usage. Judge calls are not counted;
identity, carrying the retry count, the rubric reference and its SHA-256 when one was used, and per-judge prompt versions, decoding-pin outcome and whether the response was truncated for the judge.

What is not persisted¶

The coverage guard result. insufficient_coverage is computed at scoring time and forces passed_gate to false, but it is not stored as its own column. The result screen reconstructs it from scores_json: both errored and unscored questions are missing measurements, so error_count alone cannot identify a coverage failure. The CLI has the in-memory scoring result and reports the same forced failure directly.

The must-pass failure list. must_pass_failures is returned by the runner and included in command JSON. The result page rebuilds it from the stored per-question must_pass_failed lists; it has no separate database column.

Tool-call arguments. They reach the graders and judges for the run and are never written to the result row, so they are not an audit trail. See graders.

The full response. Only the capped raw_response survives.

The judge prompts themselves. identity records which judge prompt version scored the question, not the text that was sent.

Reading a result¶

The summary band starts with N of M examples met your expectations. Here, M includes only scored examples. Unscored and errored examples are reported separately and are never folded into that headline as failures or successes. The question table puts Needs attention first so failed, unscored, and errored cases are available before cases that met expectations.

Comparing two runs¶

Choose Compare with... on a result to inspect two runs of the same test.

Direct comparison means the recorded examples, graders, must-pass graders, question and gate thresholds, and judge configuration match. The evaluated provider, model, agent, the evaluation mode (agent or chat), system prompt, token limit, temperature, and resolved model IDs may differ. Those are the inputs a direct comparison is intended to examine. A run that recorded identity drift is never a direct comparison.
Side-by-side inspection means one or more recorded evaluation conditions differ; the verdict names how many.
Inspection only means at least one run has no valid configuration snapshot, typically a result recorded before snapshots existed.

Paired examples are classified as improved, regressed, or unchanged from their composite score. If either side is unscored or errored, that pair is unscored and never counts as an improvement or regression. The comparison is descriptive only. It makes no statistical significance claim.

For must-pass comparison, a missing snapshot key equals an empty list, and list order does not matter.