Skip to content

Rubrics and deterministic checks

A rubric is a named, versioned bundle of checks that many questions can share. This page covers what a rubric holds, how a question attaches one, which check kinds the runner actually executes, and which ones exist as contract vocabulary for graders and other runners to consume. schema/rubric.schema.json and schema/README.md are authoritative.

What a rubric is

A question's criteria says what a good answer looks like for that one question. A rubric writes the same standard down once, versions it, and lets ten questions point at it.

A rubric document carries five things:

Field Required What it is
id yes Stable snake_case identifier, for example support_tone.
version yes Semver. A major bump asserts that scores from the old version are not comparable to scores from the new one.
checks yes An ordered list of at least one check.
scoring yes How the check outcomes combine, and the pass threshold for the numeric combine modes.
applicable_bundles no Which domain bundles the rubric is meaningful for. Empty means any.

A label and a description are also carried: the label is the display name in rubric lists, and the description tells an author what the rubric measures. The schema accepts both alongside the five fields above.

# yaml-language-server: $schema=https://git.drupalcode.org/project/ai_eval/-/raw/1.0.x/schema/rubric.schema.json
id: support_tone
version: "1.0"
label: "Customer-support tone"
description: "The reply should acknowledge the problem and stay polite."
checks:
  - kind: must_contain_any
    values: ["sorry", "apologies", "apologize"]
scoring:
  combine: all_pass

Attaching a rubric

A rubric attaches to a question, with a rubric_ref. There is no dataset-level attachment: a rubric applies to a whole dataset because every question in it names the same ref, which is also what lets one dataset mix standards.

questions:
  - id: Q01
    input: "A customer says their order never arrived. Reply to them."
    rubric_ref: rubric/support_tone@1.0

Both the rubric/ prefix and the @version are required. The version is what pins a question to a particular revision of the standard, so an older result stays honest about which wording produced it.

A ref resolves to an enabled rubric with that id and version. A question with no rubric_ref, or one pointing at a ref that does not resolve, is graded against its own prose criteria exactly as before.

What a resolved rubric does

Two independent things happen, and they are easy to confuse.

flowchart TD
    Q["Question with rubric_ref"]
    R["Resolved rubric"]
    TEXT["Check descriptions rendered<br/>as criteria text"]
    JUDGE["LLM judges<br/>read the merged criteria"]
    EXEC["rubric_checks grader<br/>executes checks that have an executor"]
    SCORE["Scores averaged with<br/>every other grader"]

    Q --> R
    R --> TEXT --> JUDGE --> SCORE
    R --> EXEC --> SCORE

The criteria merge always happens. At run time each check is rendered as a line of plain criteria text and appended to whatever inline criteria the question already had. That merged text is what the LLM judges read. A check's own description is used when it has one; failing that, the contains kinds, target_match, command, and score_delta render a stock sentence, and every other kind renders nothing. Opaque machine fields such as regex flags, schema refs, and judge model ids never leak into a judge prompt.

The checks are only executed if you ask for them. Deterministic execution is the rubric_checks grader's job, and it runs only when that grader is selected on the target. Attach it and the rubric's executable checks are run against the response, combined per the rubric's scoring block, and the 0 to 1 combined score is scaled onto the grader's 0 to 5 scale. A question with no executable checks scores NULL, which the scorer skips rather than counting as zero.

Running both rubric_checks and LLM judges on the same rubric scores the rubric twice, deliberately. They are two graders whose scores get averaged: one executes the checks, the other reads them as prose.

Check kinds

Every kind shares the optional base fields id, description, and metadata. Unknown fields on a check are rejected, so typos surface at validation.

The column that matters operationally is the last one. Runner-executable means the rubric_checks grader can run the check and turn it into a score. Everything else is schema-valid, is preserved intact, contributes its description to the judge criteria, and is counted in the rubric verdict's reason as not executable.

kind Required fields Key options Runner-executable
must_contain_any values case_sensitive yes
must_not_contain values case_sensitive yes
regex patterns mode (any or all), flags yes
target_match reference_path mode (exact, normalized, chrf), threshold yes
composite rubric_ref weight via the parent's scoring.weights[id] yes
json_schema one of schema_ref or schema no
php_lint wrapper, wrapper_template no
markdown_structure min_headings, max_heading_level, required_sections, no_h1, min_code_blocks, forbidden_patterns no
format format strict no
fact_match source, threshold no
tool_usage source, match_args no
llm_judge judge_prompt_ref, judges, threshold aggregation no
command command sandbox, setup, expect no
score_delta tool, min_delta sandbox, baseline, guard_command no

The kinds that run

must_contain_any, must_not_contain, and regex are substring and pattern tests over the response text. Matching is multibyte-safe and case-insensitive unless the check sets case_sensitive: true. A regex pattern is a PCRE body without delimiters, so slashes need no escaping. An invalid pattern fails the check loudly with the pattern in the reason; it is never a silent skip.

target_match compares the response against reference text named by a dot-path into the question. The path must start at expected. or metadata., for example expected.references, and may resolve to one string or an array of acceptable variants, in which case the best variant wins. normalized is the default mode: trim, collapse whitespace runs, case-fold, then compare. exact is strict equality. chrf scores character n-gram similarity and reports the continuous score, so a weighted combine sees 0.85 rather than a rounded pass or fail, while the check's own pass verdict gates on threshold. Authoring errors fail loudly: an unresolvable path, a non-scalar or empty reference, invalid UTF-8 on either side, an unknown mode, and a threshold outside 0 to 1 all fail the check.

composite reuses another rubric as one check. The sub-rubric is resolved and evaluated recursively, and its combined verdict becomes a single outcome. The recursion is cycle-guarded and capped at depth 3; a missing ref, a cycle, an over-deep chain, or a sub-rubric with nothing executable in it makes the composite check itself non-executable rather than failing the run.

The kinds that do not run

Three groups, for three different reasons.

Consumed by graders instead. llm_judge is never executed as its own judge call here. Its description folds into the criteria the LLM judges already read, and those judges produce the score. Do not treat the skip as a bug. tool_usage expresses a tool assertion as a rubric check, and no executor is planned for it: the implemented form is case-level expected_tools, scored by tool_usage_grader against the agent's own record of what it ran. See datasets. format and fact_match overlap the same way with format_grader and fact_match_grader, which read the question rather than the rubric.

Executors pending. command and score_delta are execution oracles. They need a sandbox harness, because a verification command must run inside the declared sandbox and never on the host. Progress is tracked at #3594701.

Not implemented yet. json_schema, php_lint, and markdown_structure are part of the shared contract and validate, but no executor ships for them today.

Note

A rubric made entirely of non-executable checks is not an error and produces no score from rubric_checks. Its text still reaches the judges. If you expected a deterministic number and got a skip, check which kinds you used.

Combining checks

scoring.combine is required and takes one of six values.

combine Combined score Passes when
all_pass 1.0 when every executed check passed, else 0.0 every check passed
any_pass 1.0 when at least one passed, else 0.0 at least one passed
weighted_avg weighted mean of the check scores the combined score meets scoring.threshold
min lowest check score the combined score meets scoring.threshold
max highest check score the combined score meets scoring.threshold
median median check score the combined score meets scoring.threshold

scoring.threshold is a number between 0 and 1 and applies to the four numeric modes only; omitting it defaults to 0.5. scoring.weights is keyed by check id, and a check with no id or no listed weight counts as 1.0. Weights that sum to zero score the rubric 0 and say so in the reason. The combined score is clamped to 0 through 1 before the verdict is computed.

Non-executable checks are excluded from the combine entirely rather than counted as failures. The verdict's reason names how many were skipped and which kinds they were, so a rubric that scored 1.0 on one of its five checks cannot read as a clean pass.

Authoring rubrics

Rubrics live as ai_eval_rubric content entities, at /admin/config/ai/ai-eval/rubrics under the Datasets tab. Add rubric opens the inline editor, which covers the rubric id, version, label, description, the enabled flag, the combine rule and threshold, the applicable bundles, and the checks list.

The check editor authors two kinds directly:

  • must_contain_any, which passes when the response contains any one of the values you list. Each value is a plain string, so a support-tone rubric might list "sorry", "apologize", "apologies".
  • must_not_contain, which passes when the response contains none of them.

Every other kind is shown read-only with its stored JSON preserved, so a rubric carrying richer checks is never corrupted by an edit in the browser. No YAML rubric importer ships, so those kinds have to be written into the rubric entity's data blob programmatically; the editor holds them intact from then on.

Disabling a rubric retires it without deleting it. A rubric_ref pointing at a disabled rubric resolves to nothing, and the question falls back to its own criteria.

Working examples

schema/examples/valid/ holds rubric fixtures that CI validates on every run: rubric-all-pass.yaml, rubric-weighted-avg.yaml, rubric-composite.yaml, rubric-target-match.yaml, rubric-llm-judge-single.yaml, rubric-llm-judge-panel.yaml, rubric-command.yaml, rubric-score-delta.yaml, and rubric-with-metadata.yaml.

Adding a check kind from your own module, so that it validates and executes without patching the schema, is covered in writing a check executor.