How the pieces fit together¶

The vocabulary AI Eval uses, defined once. Every other page in this section assumes these words mean what they mean here, so read this first and then follow the link to the page that owns whichever piece you need next.

The run lifecycle¶

flowchart TD
    T["Target<br/>system under test, dataset, graders, gate"]
    L["Load the dataset<br/>disabled rows dropped, optional split filter"]
    S["Snapshot the run identity<br/>provider, model, prompt and dataset hashes"]
    Q["For each question"]
    A["Agent mode<br/>invoke the ai_agents plugin"]
    C["Chat mode<br/>call the provider and model"]
    R["Resolve rubric_ref<br/>fold check text into the criteria"]
    G["Run every grader<br/>LLM judges and deterministic checks"]
    QS["Question score<br/>mean of non-null grader scores on 0 to 5"]
    TS["Target score<br/>avg_score or pass_rate"]
    GA{"Quality gate"}
    RES["Result row<br/>per-question detail, verdict, snapshot"]

    T --> L --> S --> Q
    Q --> A
    Q --> C
    A --> R
    C --> R
    R --> G --> QS
    QS --> Q
    QS --> TS --> GA --> RES

The vocabulary¶

Evaluation target¶

A target is the unit you run. It is a config entity (ai_eval_target) that ties together the system under test, the dataset of questions to ask it, the graders that score the answers, and the quality gate the run has to clear. A target is either in agent mode or chat mode, and that choice decides what "the system under test" means. See evaluation targets.

Dataset¶

A dataset is a list of questions plus an optional split rule. Datasets come from one of three sources: YAML files in a configured directory, ai_eval.dataset.* config objects, or ai_eval_dataset content entities you author in the browser. The portable file format is defined by schema/dataset.schema.json. See datasets.

Question¶

One row of a dataset, and the smallest thing that carries a score. A question needs an id and an input; everything else is optional and exists because some grader reads it. criteria is the prose standard the LLM judges grade against, expected_facts is ground truth for the fact-checking judges, expected_tools asserts what an agent did, and rubric_ref points at a shared rubric. A question marked disabled: true never runs. See datasets.

Rubric¶

A named, versioned bundle of checks with a rule for combining them, addressed as rubric/<id>@<version>. A question attaches one with rubric_ref, which is how ten questions share one standard. A resolved rubric does two things: its check descriptions are folded into the criteria text the LLM judges read, and the rubric_checks grader executes the checks that have an executor. See rubrics and checks.

Grader¶

A plugin that scores one response on one dimension and returns a score with a reason. Nine ship with the module. Six are LLM judges that call the provider configured in Settings; three are deterministic and cost no LLM calls. A target runs whichever set you select, and a grader returning NULL means "not applicable here" rather than zero. See graders.

Score¶

Graders do not all answer on the same scale, so every grader's answer is mapped onto a shared 0 to 5 composite scale before anything is averaged. A question's score is the mean of its non-null grader scores on that scale, and it passes when that mean meets the question pass threshold (3.5 by default). Individual grader scores are still displayed against their own scale, so a binary judge reads "1 / 1". See scoring and quality gates.

Quality gate¶

The pass or fail rule for the whole run. A gate names a metric, either avg_score (the mean over non-errored questions, on 0 to 5) or pass_rate (the fraction of questions that passed, on 0 to 1), a threshold, and a type. A hard gate fails the run and makes drush ai-eval:run exit 1; a soft gate logs a warning and lets the run succeed. See scoring and quality gates.

Run¶

One execution of one target against its dataset. Runs start from the Run button on the Targets page, from drush ai-eval:run, or from a scheduled CI job. A browser run advances one question per request with live progress and a cancel button; a per-target lock stops two people running the same target at once. See running evaluations.

Result¶

What a finished run leaves behind: every grader's score and reason for every question, the aggregate score, the pass or fail verdict, and a run-identity snapshot recording the configuration that produced it. The snapshot is why editing a target later does not rewrite the history of earlier runs. Results are also where you annotate individual questions and where exports read from. See sharing results.

Beyond a single run¶

Measurement is the first half. The rest of the module answers the two questions that follow a bad number.

Why does it fail? On any result you annotate individual questions as pass, fail, or defer, and tag the failure with a named mode from a taxonomy you grow as you go. Rates per mode accumulate into a chart, so a hunch becomes a number. See failure modes. At larger scale, import the OpenTelemetry spans your live site emitted, review them grouped by conversation, and promote the bad ones into a dataset so a real failure becomes a permanent regression test. See trace review.

Can I trust the scores? Most graders are LLM judges, so a model is grading a model. You label a gold set yourself, run the judge against your labels, and get a true positive rate, a true negative rate, and a trust state per judge. Trust is advisory: an untrusted judge never blocks a run, but every verdict it contributed to is marked provisional. See trusting your judges.

Once the measurement is trusted, the optimizer closes the loop. It reads a run's failures, asks an LLM for a better agent system prompt, evaluates the candidate against the same dataset, and applies it, queues it for review, or rejects it on the measured difference.

Where to start¶

Configure a judge provider and model, and check the permissions, on configuration.
Write a handful of questions: datasets.
Create a target pointing at them: evaluation targets.
Run it and read the result: running evaluations.