Evaluation targets

A target names the system under test and everything needed to score it. This page covers what a target holds, how agent mode and chat mode differ, and what each field on the target form does.

What a target is

A target is a config entity (ai_eval_target, stored as ai_eval.target.<id>). It is the only place that ties four things together:

• the system under test, which is either an ai_agents plugin or a provider and model pair;
• the dataset of questions to ask it;
• the graders that score each answer;
• the quality gate the aggregate has to clear.

Because it is config, a target exports and deploys like any other Drupal configuration, and the same target definition can move between environments. Editing a target does not rewrite the history of runs that already happened: each result carries a snapshot of the configuration in force when it started.

Targets live at Administration, Configuration, AI, AI Eval (/admin/config/ai/ai-eval). Creating, editing, and deleting one needs the Administer AI Eval permission.

The two modes

flowchart LR
    Q["Question input"]
    A["Agent mode<br/>ai_agents plugin"]
    TOOLS["Tool calls, sub-agents,<br/>agent-assembled prompt"]
    C["Chat mode<br/>provider and model"]
    SP["Target system prompt"]
    RESP["Response text<br/>plus a tool-call record in agent mode"]

    Q --> A --> TOOLS --> RESP
    Q --> C
    SP --> C
    C --> RESP

Agent mode

Agent mode invokes an AI Agents plugin end to end, the way a real user would trigger it. The agent does its own tool calls, delegates to sub-agents, and assembles its own prompt, so you are measuring the whole plugin rather than a prompt in isolation. This is the mode to use when the agent's job is to do something: read the site, create a content type, route a request.

Two consequences follow from that.

The target's own System prompt field is not used in agent mode. The prompt that reaches the model is assembled inside ai_agents and is not part of what ai_eval records. The optimizer works on agent mode by editing the system_prompt of the agent's config entity, not the target's.

Most graders read only the response text, which in agent mode measures how the answer reads rather than whether the work happened. The Targets list marks an agent-mode target whose graders can all only read text with a Text only badge, and drush ai-eval:run says the same thing. Attach tool_usage_grader with expected_tools on your questions to score behavior. See graders.

Agent mode needs the ai_agents module installed. Without it the Agent ID field degrades to a plain text field.

Chat mode

Chat mode sends the question straight to any provider the AI module supports, with the target's system prompt prepended. No agent framework is involved, so what you are measuring is a system prompt, a model, and a decoding configuration. Use it for system prompts, retrieval-augmented pipelines that expose a chat endpoint, question-and-answer bots, and classification prompts.

Chat mode has no tools and therefore no tool-call record, so expected_tools assertions fall back to reading the response text, and the two judges that would otherwise be shown the record are shown nothing.

Both modes share the same datasets, graders, scoring, gates, and reporting.

Creating a target

1. Go to /admin/config/ai/ai-eval.
2. Use the Add eval target action (/admin/config/ai/ai-eval/targets/add).
3. Fill in the fields below and save.

Clicking a target's name afterwards opens its workflow hub (/admin/config/ai/ai-eval/targets/<id>/workflow), which recommends the next step depending on whether the target still needs cases, needs error analysis, or is ready to operate and optimize.

Field reference

Field	Stored as	Notes
Label	label	Required. Display name.
Machine name	id	Required, fixed after the first save.
Evaluation mode	mode	agent or chat. Decides which of the two sections below applies.
Agent ID	agent_id	Agent mode only. A select of installed ai_agents plugins, or a text field when ai_agents is absent.
Provider / Model	provider_id, model_id	Chat mode only. A select of the AI module's chat provider and model pairs. If the provider list cannot be loaded it falls back to a text field expecting provider__model, for example openai__gpt-4o-mini.
System prompt	system_prompt	Chat mode only. Prepended to every question. Ignored in agent mode.
Max response tokens	max_tokens	Chat mode only. Empty means the provider default. A modest cap lowers cost and avoids upfront affordability errors on metered providers.
Sampling temperature	temperature	Chat mode only, 0 to 2. Empty means the provider default, which is normally what you want: the target is the system under test and should run at its production temperature.
Dataset	dataset_source, dataset_ref	Required. One grouped select, with an optgroup per source (File, Config, Entity). The chosen option is split into the source plugin id and the reference within it.
Graders	graders	Checkboxes in two groups, LLM judges and deterministic checks. Each label carries the scale that grader answers on. Every selected grader scores every question.
Question pass threshold	question_pass_threshold	0 to 5. Empty uses the global default from Settings.
Response char limit	response_char_limit	Minimum 100. Caps the response length format_grader scores against. Empty uses the global default. LLM judges are not bounded by it; they are bounded separately by judge_response_char_limit.
Gate threshold	threshold	Required. Measured on the metric's own scale: 0 to 5 for avg_score, 0 to 1 for pass_rate.
Gate metric	metric	avg_score (Average score) or pass_rate (Pass rate).
Gate type	gate	Hard fails the run and exits 1; soft logs a warning and lets the run succeed.
Proving-ground fixtures	fixtures	Optional, one name@version pin per line, for example kb_docs@1. Names the fixtures an eval bed provisions and resets for this target. Leave empty when the target does not run against a proving-ground environment.

The gate threshold field carries a live note reading / 5 ceiling. That ceiling is the shared composite scale, not the scale of any one grader: every grader is mapped onto 0 to 5 before scores are averaged, so a target graded only by a binary judge is still gated on 0 to 5, and 3.5 is a sensible threshold for it.

What the form refuses

Four checks run on save.

• A pass_rate threshold above 1.0 is rejected. A pass rate cannot exceed 100 percent, and the avg_score default of 4.0 left behind after a metric switch is the usual way to hit this.
• An avg_score threshold above 5.0 is rejected as an unpassable gate. No run could ever reach it.
• A malformed or duplicated fixture pin is rejected, so a corrupt binding never reaches the entity.
• A manually typed provider and model without the __ separator is rejected.

One case produces a warning rather than an error: an avg_score threshold between 0 and 1 saves, but the form points out that such a threshold passes almost any run and asks whether you meant pass_rate.

Reading the Targets list

The landing page is a portfolio view. Each row carries the gate, the latest average score against its threshold, the pass rate, the trust state of the judges that target uses, when it last ran, how many cases it holds, and a Run button.

Two badges can appear next to the trust state:

• Text only, on an agent-mode target with no grader that can observe what the agent did.
• Judge same family, when the configured judge shares a coarse model family with the target's own model. Models prefer their own family's output, so part of every score is the judge grading its own reflection. This marker is row-level only for chat-mode targets, because an agent-mode target exposes no model id in config; for those the comparison is reported per run instead. See trusting your judges.

Neither badge blocks anything. Both qualify what the numbers next to them mean.