Configuration

What to set up after drush en ai_eval: who can do what, where the judge model comes from, and what every field on the settings form controls. If the module is not installed yet, start at installation.

Permissions

AI Eval declares four permissions. Grant them at /admin/people/permissions.

Machine name	Label	Grants
operate ai eval	Operate AI Eval	View results, traces and dashboards, browse eval targets and optimization candidates, and run evaluations.
administer ai eval	Administer AI Eval	Manage settings, add, edit, and delete eval targets, and apply or reject optimization candidates.
annotate ai eval results	Annotate AI Eval results	Annotate run results and label traces, and promote them into datasets.
validate ai eval judges	Validate AI Eval judges	Validate LLM judges against human-labeled examples, edit judge prompts, and view per-grader trust state. Label traces for judge validation.

How the routes combine them

Read the routing before you design roles around these. The four permissions map onto route groups like this:

Route group	Requirement
/admin/config/ai/ai-eval/manage	operate ai eval or administer ai eval
/admin/config/ai/ai-eval/tests/new/{step}, /admin/config/ai/ai-eval/targets/{target}/edit-guided/{step}	administer ai eval
/admin/config/ai/ai-eval/results/{a}/compare/{b}	operate ai eval or administer ai eval
Settings, target and dataset add, edit, delete, rubric management, failure-mode add, edit and delete, question manager, optimizer apply and reject	administer ai eval
Targets list, results, target dashboards, run start, progress and cancel, trace review and import, optimizer candidate views	operate ai eval or administer ai eval
Trace labeling (individual traces and conversation or agent-run verdicts)	annotate ai eval results, validate ai eval judges or administer ai eval
Promoting traces into datasets	annotate ai eval results or administer ai eval
Result annotation, annotation promotion and reset	annotate ai eval results or administer ai eval
Judge validation, judge prompt editing, gold labeling	validate ai eval judges

For example, the routing file uses operate ai eval+administer ai eval and annotate ai eval results+administer ai eval. In Drupal a + between permissions on a route means or, and a comma means and, so an administrator reaches those screens without also being granted Operate or Annotate. That is the intent: Operate and Annotate exist so you can build a reviewer role that does not administer the site.

Note

Opening a screen and saving from it require the permissions for each route. Trace review needs Operate or Administer to view it. Gold labeling needs Validate AI Eval judges, which also permits saving labels. Promoting from trace review or gold labeling needs Annotate AI Eval results or Administer AI Eval. Trusting your judges covers the labeling case.

AI provider setup

AI Eval configures no providers and stores no API keys. It uses the providers you have already set up in the AI module, at /admin/config/ai/providers. Configure at least one provider that supports the chat operation type before you go further.

Two things read that provider list:

• The judge, set once on the AI Eval settings form and used by every LLM grader. The judge dropdown is populated from the AI module's provider and model options for the chat operation.
• Chat-mode targets, which pick their own provider and model per target. See targets.

If the provider list cannot be loaded (no provider is configured, or a provider's model-list call fails), the settings form does not white-screen. It logs a warning, shows it to you, and degrades the judge and proposer fields to plain text inputs that take a provider__model string such as openai__gpt-4o-mini. Two underscores, not one.

The settings form

/admin/config/ai/ai-eval/settings, or Manage > Settings under Administration, Configuration, AI, AI Eval. Requires administer ai eval. Everything lands in the ai_eval.settings config object, and changes take effect immediately with no cache rebuild.

LLM judge

Field	Config keys	Default	What it does
Judge provider / model	judge_provider, judge_model	empty	The LLM every judge grader calls. Deterministic graders ignore it. Required.
Prompt proposer provider / model	proposer_provider, proposer_model	empty	The model that writes candidate prompts. Empty means reuse the site judge model (not a per-judge override). A judge from the same model family tends to prefer its own family's prompts.
Do not count a pass that rests on a judge that is not trusted	refuse_unvalidated_judges	false	Record a gate pass as failed when it rests on a judge that is not trusted, and name those judges on the result. Applies to runs that finish after the change; earlier results keep their verdict.
Pin judges to temperature 0	pin_judge_decoding	true	Ask LLM judges for temperature 0 and top_p 1 when offered by the provider. If the model refuses, retry without the settings and record that it was not pinned. Turn off to use provider defaults.

A fresh install ships both judge keys empty, which is why the status report flags AI Eval as misconfigured until you set them. The form preselects no model in that state, and no vendor is used in its place: until a judge has a model (the site judge, or its own on the judge's Edit prompt tab), it scores nothing and records each answer as an error, No judge model is set. Choose one in AI Eval settings or on the judge's Edit prompt tab. The Judges screen shows the same sentence under the judge's name, and the judge's validation page shows it in place of the launch form. A content review judge with no model reviews nothing either: it does not fall back to the review rule's own model. You can still add such a judge to a test; it scores once it has a model.

Set the proposer to a different model family from the judge. When one model both writes a candidate prompt and scores it, the score carries self-preference bias. The field description says the same thing; it is left optional so an evaluation with a single configured provider still runs.

Scoring

Field	Config key	Default	Range	What it does
Default question pass threshold	question_pass_threshold	3.5	0 to 5	Minimum average score for a question to count as passed. A target can override it.
Default response char limit	response_char_limit	4000	100 or more	Maximum stored response length before format_grader penalizes. A target can override it.
Judge response char limit	judge_response_char_limit	4000	100 or more	Responses longer than this are truncated before going to the LLM judge, which bounds judge context and cost.

The two character limits are separate on purpose. response_char_limit bounds what format_grader and trace sampling read. LLM judges are not subject to it; they are bounded by judge_response_char_limit. See scoring.

Optimizer

Field	Config key	Default	What it does
Require review before applying	require_review	true	Store optimized prompts as pending for review instead of applying them.
Improvement margin	improvement_margin	0.4	Minimum measured score improvement before a candidate prompt is accepted.
Max optimization attempts	max_optimization_attempts	3	Prompt optimization iterations per target.
Candidate runs per attempt	optimizer_candidate_runs	3	How many times each candidate and the baseline are re-evaluated, to average out judge noise.
Minimum dataset size	optimizer_min_dataset_size	15	The optimizer refuses targets with fewer questions than this, so it does not overfit judge noise.
Sample-size confidence level	cochran_confidence	0.95	How sure the sample-size estimate should be.
Pass-rate margin of error	cochran_margin_of_error	0.05	How much the measured pass rate may differ from the underlying rate.
Optimizer guidelines	optimizer_guidelines	empty	Optional free-text guidance folded into the optimizer metaprompt.

The two sample-size fields drive advisory warnings only. They never change a gate verdict or a Drush exit code. See optimizer.

Paths

Field	Config key	Default	What it does
Dataset path	dataset_path	empty	Directory scanned first for dataset YAML files. The module's own data/ and each enabled module's datasets/ directory are also scanned. The first matching filename wins.
Results path	results_path	empty	Directory for optional JSON result exports. Empty means an ai_eval_results directory inside the system temp directory.

Note

Uninstall removes only top-level results-YYYYMMDD-HHMMSS.json files and envelopes/*.eee.json in the configured results path, then the envelopes folder if empty. Other files and the results path stay. The default temporary folder and exports made with --output-dir are not removed.

Rate limiting

Field	Config key	Default	What it does
Max retries	rate_limit_max_retries	3	Retry count on provider rate-limit errors.
Base delay (seconds)	rate_limit_base_delay	5.0	Seconds to wait before the first retry after a rate-limit error.
Delay between calls (seconds)	rate_limit_delay_between_calls	2.0	Throttle between consecutive API calls.
Run interruption window	run_stale_seconds	300	Shows an active browser-driven run as interrupted after this many seconds without progress. The minimum is 30 seconds.

A full run can mean hundreds of provider calls. The retry and delay settings help keep it within a provider quota.

Judge trust, gold labels, and export

Set the judge validation age limit under LLM judge, the gold label target under Gold labels, and automatic export under Export on the settings form. The age limit is a whole number of days, 0 or more (0 treats every validation as out of date). The gold label target is a whole number of 1 or more. You can also set these values with Drush or in exported config.

Config key	Default	What it does
judge_trust_stale_days	30	Days after which a trusted judge validation decays to stale. Evaluated at read time, so changing it re-rates existing reports with no migration.
export_envelope_on_complete	false	Export an EEE envelope automatically when a run completes. See sharing results.
gold_target_n	30	Gold labels per judge dimension that the labeling screen counts coverage against; zero also uses 30. See judges.

drush config:set ai_eval.settings export_envelope_on_complete true
drush config:set ai_eval.settings judge_trust_stale_days 14

Fresh installs store a 30-day judge validation age limit. The update adds 30 only when the setting is missing; existing values are preserved.

Content review judge

With the ai_eval_content_review submodule, an AI Content Review criterion is a judge. Set it up at Manage > Content review judge (/admin/config/ai/ai-eval/content-review, administer ai eval):

Field	What it sets
Review rule	The rule editors use. Disabled rules are marked.
Criterion	The criterion of that rule the base judge reviews with.
Content type	The type of the unsaved page each answer becomes.
Body field	The formatted text field the answer goes into. Only text fields of the chosen type are offered.
Text format	The format of that body. Only formats the field allows are offered.
Other field values	field_name|value lines for other fields the criterion reads.

The form refuses settings that do not fit the site, such as a body field the content type lacks. The status report shows the same check, and a judge with such settings reports its answers as unmeasured instead of scoring them. A new install starts with no content type, body field or format: choose them before the judge can run. The submodule's README covers validating a criterion.

Checking your work

/admin/reports/status carries three AI Eval checks: judge configuration, dataset path validity, and whether any target exists. Clear the first two, then create a target and run it. See running evaluations.