Installation

How to install AI Eval, what it depends on, and which extras you need for which features. For what to set up once the module is enabled, see configuration.

Quick install

composer require drupal/ai_eval
drush en ai_eval

Composer pulls the AI module and the opis/json-schema library with it. Drush enables the two core modules AI Eval declares, Options and File, if they are not already on.

Install as you would any contributed Drupal module. See Installing Drupal Modules if you want the long version.

Requirements

Requirement	Constraint	Where it is declared
Drupal core	^11.3	composer.json, ai_eval.info.yml
PHP	8.3 or later	ai_eval.info.yml
AI (drupal/ai)	^1.0	composer.json (ai_eval.info.yml lists it without a constraint)
opis/json-schema	^2.3	composer.json
Options (core)	enabled with the module	ai_eval.info.yml
File (core)	enabled with the module	ai_eval.info.yml

opis/json-schema is a hard runtime requirement, not a dev-only one. AI Eval uses it to validate rubrics, and to validate datasets loaded from the config and entity sources, against the JSON Schemas in schema/. It is needed on production sites, not just in test runs. Composer installs it for you; you do not enable anything.

Warning

Do not download AI Eval as a tarball and drop it into modules/. Without Composer the opis/json-schema library is missing and rubric and dataset validation will fatal.

Not everything is validated

Validation runs where something calls a validator, which is narrower than the schemas suggest. Datasets loaded from a file are not validated: the file source parses the YAML and hands the rows on. Failure-mode taxonomy imports are not validated either; that validator exists as a service but nothing calls it. See datasets and failure modes.

Optional dependencies

Composer declares two suggestions. Neither is installed automatically.

# Agent mode
composer require drupal/ai_agents

# The ai_eval_droost submodule
composer require drupal/droost

AI Agents, for agent mode

AI Agents is needed only to evaluate agent plugins end to end. Chat mode, which prompts a provider directly, works without it, and so does everything downstream: datasets, graders, scoring, judge validation, trace review, and the optimizer.

flowchart TD
    CORE["ai_eval"]
    AI["drupal/ai<br/>required"]
    OPIS["opis/json-schema<br/>required"]
    AGENTS["drupal/ai_agents<br/>agent mode only"]
    DROOST["drupal/droost<br/>ai_eval_droost only"]

    CORE --> AI
    CORE --> OPIS
    CORE -.-> AGENTS
    CORE -.-> DROOST

Droost, for the state grader submodule

The module ships one submodule, ai_eval_droost, which adds a deterministic grader that ignores the response text and asserts real Drupal state through Droost read and validate tools. It depends on droost and will not enable without it.

drush en ai_eval_droost

Trace data producers

Trace review reads OTLP GenAI span JSON exported from a live site. AI Eval parses those files; it does not depend on the modules that write them. To produce the spans you need the AI module's ai_observability submodule plus the OpenTelemetry module, on whichever site you are observing. Both are optional and every other feature works without them.

After enabling

AI Eval registers runtime status checks, so the status report at /admin/reports/status tells you what is still missing:

Check	Fails when
AI Eval judge configuration	The judge provider or judge model is unset
AI Eval dataset path	The configured dataset directory does not exist
AI Eval targets	No eval target has been created yet (warning only)

A fresh install fails the judge check, because the shipped defaults for judge_provider and judge_model are empty. Set them before you run anything. Configuration covers that, along with permissions and how AI Eval picks up the providers configured in the AI module.

Uninstalling

drush pmu ai_eval

Uninstall deletes every .json file under the configured results path, if one is set. It does not touch datasets you authored as YAML on disk.

Warning

The deletion scans the results path recursively and matches on the .json extension alone. Nothing records which files AI Eval wrote, so a results path pointing at a shared directory loses unrelated JSON too. Give it a directory of its own.