Configuration

What to set up after drush en ai_eval: who can do what, where the judge model comes from, and what every field on the settings form controls. If the module is not installed yet, start at installation.

Permissions

AI Eval declares four permissions. Grant them at /admin/people/permissions.

Machine name	Label	Grants
operate ai eval	Operate AI Eval	View results and dashboards, browse eval targets and optimization candidates, and run evaluations.
administer ai eval	Administer AI Eval	Manage settings, add, edit, and delete eval targets, and apply or reject optimization candidates.
annotate ai eval results	Annotate AI Eval results	Annotate eval-run question outcomes (pass, fail, defer, plus notes) for error analysis, and promote annotations into datasets.
validate ai eval judges	Validate AI Eval judges	Validate LLM judges against human-labeled examples, edit judge prompts, and view per-grader trust state.

How the routes combine them

Read the routing before you design roles around these. The four permissions map onto route groups like this:

Route group	Requirement
Settings, target and dataset add, edit, delete, rubric management, failure-mode add, edit and delete, question manager, optimizer apply and reject	administer ai eval
Targets list, results, target dashboards, run start, progress and cancel, trace labeling and promotion, optimizer candidate views	operate ai eval or administer ai eval
Result annotation, annotation promotion and reset	annotate ai eval results or administer ai eval
Judge validation, judge prompt editing, gold labeling	validate ai eval judges

The routing file writes the middle two as operate ai eval+administer ai eval and annotate ai eval results+administer ai eval. In Drupal a + between permissions on a route means or, and a comma means and, so an administrator reaches those screens without also being granted Operate or Annotate. That is the intent: Operate and Annotate exist so you can build a reviewer role that does not administer the site.

Note

A handful of actions are gated a second time inside the controller rather than on the route. Gold labeling, for example, is a validate ai eval judges route, but the buttons that write a label also check for Operate or Administer. Opening a screen and being able to save from it are separate questions. Trusting your judges covers the labeling case.

AI provider setup

AI Eval configures no providers and stores no API keys. It uses the providers you have already set up in the AI module, at /admin/config/ai/providers. Configure at least one provider that supports the chat operation type before you go further.

Two things read that provider list:

• The judge, set once on the AI Eval settings form and used by every LLM grader. The judge dropdown is populated from the AI module's provider and model options for the chat operation.
• Chat-mode targets, which pick their own provider and model per target. See targets.

If the provider list cannot be loaded (no provider is configured, or a provider's model-list call fails), the settings form does not white-screen. It logs a warning, shows it to you, and degrades the judge and proposer fields to plain text inputs that take a provider__model string such as openai__gpt-4o-mini. Two underscores, not one.

The settings form

/admin/config/ai/ai-eval/settings, or the Settings tab under Administration, Configuration, AI, AI Eval. Requires administer ai eval. Everything lands in the ai_eval.settings config object, and changes take effect immediately with no cache rebuild.

LLM judge

Field	Config keys	Default	What it does
Judge provider / model	judge_provider, judge_model	empty	The LLM every judge grader calls. Deterministic graders ignore it. Required.
Prompt proposer provider / model	proposer_provider, proposer_model	empty	The LLM the optimizer uses to write candidate prompts. Empty means reuse the judge model.

A fresh install ships both judge keys empty, which is why the status report flags AI Eval as misconfigured until you set them.

Set the proposer to a different model family from the judge. When one model both writes a candidate prompt and scores it, the score carries self-preference bias. The field description says the same thing; it is left optional so an evaluation with a single configured provider still runs.

Scoring

Field	Config key	Default	Range	What it does
Default question pass threshold	question_pass_threshold	3.5	0 to 5	Minimum average score for a question to count as passed. A target can override it.
Default response char limit	response_char_limit	4000	100 or more	Maximum stored response length before format_grader penalizes. A target can override it.
Judge response char limit	judge_response_char_limit	4000	100 or more	Responses longer than this are truncated before going to the LLM judge, which bounds judge context and cost.

The two character limits are separate on purpose. response_char_limit bounds what format_grader and trace sampling read. LLM judges are not subject to it; they are bounded by judge_response_char_limit. See scoring.

Optimizer

Field	Config key	Default	What it does
Require review before applying	require_review	true	Store optimized prompts as pending for review instead of applying them.
Improvement margin	improvement_margin	0.4	Minimum measured score improvement before a candidate prompt is accepted.
Max optimization attempts	max_optimization_attempts	3	Prompt optimization iterations per target.
Candidate runs per attempt	optimizer_candidate_runs	3	How many times each candidate and the baseline are re-evaluated, to average out judge noise.
Minimum dataset size	optimizer_min_dataset_size	15	The optimizer refuses targets with fewer questions than this, so it does not overfit judge noise.
Cochran confidence level	cochran_confidence	0.95	Confidence level used when reporting sample-size warnings.
Cochran margin of error	cochran_margin_of_error	0.05	Target margin of error used when reporting sample-size warnings.
Optimizer guidelines	optimizer_guidelines	empty	Optional free-text guidance folded into the optimizer metaprompt.

The two Cochran fields drive advisory warnings only. They never change a gate verdict or a Drush exit code. See optimizer.

Paths

Field	Config key	Default	What it does
Dataset path	dataset_path	empty	Directory scanned for dataset YAML files. Empty means the module's own data/ directory.
Results path	results_path	empty	Directory for optional JSON result exports. Empty means an ai_eval_results directory inside the system temp directory.

Note

Uninstalling AI Eval deletes every .json file in the configured results path. Point it at a directory AI Eval owns, not a shared one.

Rate limiting

Field	Config key	Default	What it does
Max retries	rate_limit_max_retries	3	Retry count on provider rate-limit errors.
Base delay (seconds)	rate_limit_base_delay	5.0	Initial backoff delay.
Delay between calls (seconds)	rate_limit_delay_between_calls	2.0	Throttle between consecutive API calls.

A full run can mean hundreds of provider calls. These three settings are what stops it tripping a provider quota.

Settings with no form field

Three keys in ai_eval.settings have no UI. Set them with Drush or in exported config.

Config key	Default	What it does
judge_trust_stale_days	30	Days after which a trusted judge validation decays to stale. Evaluated at read time, so changing it re-rates existing reports with no migration.
export_envelope_on_complete	false	Export an EEE envelope automatically when a run completes. See sharing results.
gold_target_n	30	Gold labels per judge dimension that the labeling screen counts coverage against. See judges.

drush config:set ai_eval.settings export_envelope_on_complete true
drush config:set ai_eval.settings judge_trust_stale_days 14

judge_trust_stale_days is not written by the install profile, so a fresh site has no value for it and the code default of 30 applies until you set one.

Checking your work

/admin/reports/status carries three AI Eval checks: judge configuration, dataset path validity, and whether any target exists. Clear the first two, then create a target and run it. See running evaluations.