AI Eval

AI Eval is an evaluation framework for AI features built on Drupal. You describe what a good answer looks like, AI Eval asks your agent or model the questions, scores every answer with a mix of LLM judges and deterministic checks, and tells you whether the run cleared the quality gate you configured. Over time it also helps you work out why a system fails, prove that the automated scoring can be trusted, and improve the prompts that drive it.

The problem it solves

Teams building on AI hit the same question sooner or later: is this thing actually getting better? Reading a handful of responses by hand does not scale and does not catch regressions. You fix one answer and quietly break three others.

AI Eval answers that question with data. It runs a fixed set of questions (a dataset) against your AI system, scores every response with a set of graders, aggregates those scores, and compares the result against a quality gate. The run either passes or fails, and drush ai-eval:run exits non-zero on a failed hard gate, so the verdict works in a CI pipeline.

The loop

flowchart TD
    DS["Dataset<br/>questions, criteria, rubrics"]
    RE["Run engine"]
    AG["Agent mode<br/>ai_agents plugin"]
    CH["Chat mode<br/>AI provider and model"]
    GR["Graders<br/>LLM judges and deterministic checks"]
    SC["Composite score<br/>avg_score and pass_rate"]
    QG{"Quality gate"}
    OK["Pass, recorded as a result"]
    OPT["Optimizer<br/>proposes a new system prompt"]

    DS --> RE
    RE --> AG
    RE --> CH
    AG --> GR
    CH --> GR
    GR --> SC
    SC --> QG
    QG -->|"meets threshold"| OK
    QG -->|"below threshold"| OPT
    OPT -->|"re-evaluate the candidate"| RE

Everything in AI Eval belongs to that loop, and each admin screen is one stop on it. A target names the system under test: which agent or which provider and model, which dataset, which graders, and what gate the run must clear. Running a target produces a result: one score per grader per question, an aggregate score, and a pass or fail verdict.

Two evaluation modes

Agent mode evaluates AI Agents plugins end to end, the way a real user would trigger them. The agent does its own tool calls and assembles its own prompt, so you are measuring the whole plugin, not a prompt in isolation.

Chat mode sends prompts straight to any provider supported by the AI module, so you can evaluate a model and prompt combination with no agent framework in the picture.

Both modes share the same datasets, graders, scoring, and reporting. See targets for how to pick between them.

What is in the box

• Targets and modes. One config entity ties the system under test to its dataset, graders, and gate. See targets.
• Datasets. Questions live as YAML files on disk, as content entities authored in the UI, or as config shipped inside a module. See datasets.
• Rubrics. Named, versioned bundles of checks that many questions can share by reference. See rubrics.
• Graders. Nine grader plugins ship with the module, covering accuracy, relevance, completeness, actionability, fact match, groundedness, format, tool usage, and deterministic rubric checks. See graders.
• Running. Start a run from the Targets page, from Drush, or from a scheduled CI job. See running evaluations.
• Scoring and gates. Per-grader scores map onto a shared 0 to 5 composite scale, and the gate checks either avg_score or pass_rate against a threshold. See scoring.
• Judge validation. Label a gold set yourself, run each LLM judge against your labels, and get TPR, TNR, and a trust state per judge. See judges.
• Trace review. Import OpenTelemetry GenAI spans from a live site, review them grouped by conversation, label them, and promote the bad ones into datasets. See traces.
• Failure modes. A taxonomy of named failure codes you attach to annotations, with per-mode rates over time. See failure modes.
• Optimizer. Reads the failures from a run, asks an LLM to propose a better system prompt, evaluates the candidate against the same dataset, and applies, proposes, or rejects it on the measured improvement. See optimizer.
• Portable exports. Completed runs serialize to EEE JSON artifacts for cross-site comparison. See sharing results.

The admin UI lives under Administration, Configuration, AI, AI Eval (/admin/config/ai/ai-eval), as one menu entry with six tabs: Targets, Datasets, Traces, Judges, Results, and Settings. It is server-rendered and made interactive with htmx, so a long eval run advances one question per request with live progress and a working cancel button.

Note

Most graders are LLM judges, which means a model is grading a model. Judge trust is advisory and never blocks a run, but any verdict that an untrusted, stale, or never-validated judge contributed to is marked provisional everywhere it appears. Judge validation is how you clear that flag.

Where to go next

• Installation: composer, drush, and the dependency picture.
• Usage: how datasets, graders, scores, and gates fit together.
• Developers: the plugin seams, events, and JSON Schemas.