Installation

How to install AI Eval, what it depends on, and which extras you need for which features. For what to set up once the module is enabled, see configuration.

Quick install

composer require 'drupal/ai_eval:^1.1@alpha'
drush en ai_eval

The 1.1.x branch is in alpha and adds workflow targets and decision calibration. The 1.0.x branch is in beta; install it with composer require 'drupal/ai_eval:~1.0.0@beta'. Neither branch has a stable release yet, so name the version.

Composer pulls the AI module and the opis/json-schema library with it. Drush enables the two core modules AI Eval declares, Options and File, if they are not already on.

Install as you would any contributed Drupal module. See Installing Drupal Modules if you want the long version.

Requirements

Requirement	Constraint	Where it is declared
Drupal core	^11.3	composer.json, ai_eval.info.yml
PHP	8.3 or later	ai_eval.info.yml
AI (drupal/ai)	^1.0	composer.json (ai_eval.info.yml lists it without a constraint)
opis/json-schema	^2.3	composer.json
Options (core)	enabled with the module	ai_eval.info.yml
File (core)	enabled with the module	ai_eval.info.yml

opis/json-schema is a hard runtime requirement, not a dev-only one. AI Eval uses it to validate rubrics, and to validate datasets loaded from the config and entity sources, against the JSON Schemas in schema/. It is needed on production sites, not just in test runs. Composer installs it for you; you do not enable anything.

Warning

Do not download AI Eval as a tarball and drop it into modules/. Without Composer the opis/json-schema library is missing and rubric and dataset validation will fatal.

Not everything is validated

File, config, and entity dataset sources validate documents before loading questions. Failure-mode taxonomy imports are not validated; that validator exists as a service but nothing calls it. See datasets and failure modes.

Optional dependencies

Composer declares six suggestions: drupal/search_api, drupal/token, drupal/ai_agents, drupal/droost, drupal/tool, and drupal/ai_content_review. None is installed automatically. drupal/ai_answers is another optional integration; it is not a Composer suggestion.

# Agent mode
composer require drupal/ai_agents

# The ai_eval_droost submodule (Droost 2 and Tool API). -W lets twig move
# to 3.29: Droost 2.0.0 conflicts with twig 3.30.0.
composer require -W 'drupal/droost:^2.0' 'drupal/tool:^1.0@beta'

# The ai_eval_search_api submodule
composer require drupal/search_api

# The ai_eval_ai_automators submodule
composer require drupal/token

# The ai_eval_content_review submodule
composer require drupal/ai_content_review

# The ai_eval_ai_answers submodule
composer require drupal/ai_answers

AI Agents, for agent mode

AI Agents is needed only to evaluate agent plugins end to end. Chat mode, which prompts a provider directly, works without it, and so does everything downstream: datasets, graders, scoring, judge validation, trace review, and the optimizer.

flowchart TD
    CORE["ai_eval"]
    AI["drupal/ai<br/>required"]
    OPIS["opis/json-schema<br/>required"]
    AGENTS["drupal/ai_agents<br/>agent mode only"]
    DROOST["drupal/droost 2 + drupal/tool<br/>ai_eval_droost only"]
    ANSWERS["drupal/ai_answers<br/>ai_eval_ai_answers only"]

    CORE --> AI
    CORE --> OPIS
    CORE -.-> AGENTS
    CORE -.-> DROOST
    CORE -.-> ANSWERS

Droost, for the state grader submodule

The ai_eval_droost submodule adds a deterministic grader that ignores the response text and asserts real Drupal state through Droost read and validate tools, called through Tool API. It depends on Droost 2.0.0 or later and on Tool API (tool), and will not enable without them. View specs grade from Drush (drush aer) only; in the admin UI they fail with a reason that says so. The grader's tools live in Droost submodules: enable droost_build for View specs and droost_canvas for Canvas specs.

drush en ai_eval_droost droost_build
# Only if you grade Canvas pages:
drush en droost_canvas

AI Answers, for grading search answers

The ai_eval_ai_answers submodule runs an AI Answers agent as a workflow target, so the answers visitors get on search can be graded (see workflow mode). It depends on ai_answers and will not enable without it.

drush en ai_eval_ai_answers

Trace data producers

Trace review reads OTLP GenAI span JSON exported from a live site. AI Eval parses those files; it does not depend on the modules that write them. To produce the spans you need the AI module's ai_observability submodule plus the OpenTelemetry module, on whichever site you are observing. Both are optional and every other feature works without them.

After enabling

AI Eval registers runtime status checks, so the status report at /admin/reports/status tells you what is still missing:

Check	Fails when
AI Eval judge configuration	The judge provider or judge model is unset
AI Eval dataset path	The configured dataset directory does not exist
AI Eval targets	No eval target has been created yet (warning only)

A fresh install fails the judge check, because the shipped defaults for judge_provider and judge_model are empty. Set them before you run anything. Configuration covers that, along with permissions and how AI Eval picks up the providers configured in the AI module.

Uninstalling

drush pmu ai_eval

Uninstall removes the results-YYYYMMDD-HHMMSS.json files AI Eval wrote at the top level of the configured results path, and the *.eee.json files directly in its envelopes folder. The envelopes folder is removed if empty; other files and the results path stay. Files in the default temporary folder and files exported with ai-eval:export-envelope --output-dir are not removed. Datasets you authored as YAML on disk stay too.

Note

Give the results path a directory of its own anyway, so the files are easy to find.