Evaluation targets¶

A target names the system under test and everything needed to score it. This page covers what a target holds, how agent, chat and workflow mode differ, and what each field on the target form does.

What a target is¶

A target is a config entity (ai_eval_target, stored as ai_eval.target.<id>). It is the only place that ties four things together:

the system under test, which is an ai_agents plugin, a provider and model pair, or a workflow in an engine that has registered a runner;
the dataset of questions to ask it;
the graders that score each answer;
the quality gate the aggregate has to clear.

Because it is config, a target exports and deploys like any other Drupal configuration, and the same target definition can move between environments. Editing a target does not rewrite the history of runs that already happened: each result carries a snapshot of the configuration in force when it started.

Targets live at Administration, Configuration, AI, AI Eval (/admin/config/ai/ai-eval). Creating, editing, and deleting one needs the Administer AI Eval permission.

The three modes¶

flowchart LR
    Q["Question input"]
    A["Agent mode<br/>ai_agents plugin"]
    TOOLS["Tool calls, sub-agents,<br/>agent-assembled prompt"]
    C["Chat mode<br/>provider and model"]
    SP["Target system prompt"]
    W["Workflow mode<br/>engine runner"]
    STEPS["Pipeline steps and<br/>model calls"]
    RESP["Response text<br/>plus a tool-call record in agent mode<br/>plus a step record in workflow mode"]

    Q --> A --> TOOLS --> RESP
    Q --> C
    SP --> C
    C --> RESP
    Q --> W --> STEPS --> RESP

Agent mode¶

Agent mode invokes an AI Agents plugin end to end, the way a real user would trigger it. The agent does its own tool calls, delegates to sub-agents, and assembles its own prompt, so you are measuring the whole plugin rather than a prompt in isolation. This is the mode to use when the agent's job is to do something: read the site, create a content type, route a request.

Two consequences follow from that.

The target's own System prompt field is not used in agent mode. The prompt that reaches the model is assembled inside ai_agents and is not part of what ai_eval records. The optimizer works on agent mode by editing the system_prompt of the agent's config entity, not the target's.

Most graders read only the response text, which in agent mode measures how the answer reads rather than whether the work happened. The Targets list marks an agent-mode target whose graders can all only read text with a Text only badge, and drush ai-eval:run says the same thing. Attach tool_usage_grader with expected_tools on your questions to score behavior. See graders.

Agent mode needs the ai_agents module installed. Without it the Agent ID field degrades to a plain text field.

Chat mode¶

Chat mode sends the question straight to any provider the AI module supports, with the target's system prompt prepended. No agent framework is involved, so what you are measuring is a system prompt, a model, and a decoding configuration. Use it for system prompts, retrieval-augmented pipelines that expose a chat endpoint, question-and-answer bots, and classification prompts.

Chat mode has no tools and therefore no tool-call record, so expected_tools assertions fall back to reading the response text, and the two judges that would otherwise be shown the record are shown nothing.

Workflow mode¶

Workflow mode puts a pipeline under test: deterministic steps around one or more model calls, run by a workflow engine such as FlowDrop. ai_eval does not know any engine. The engine, or a bridge module for it, implements \Drupal\ai_eval\Workflow\WorkflowRunnerInterface and registers the implementation as a service tagged ai_eval.workflow_runner; the target names that runner by its engine id. With no runner installed, a workflow target is refused at run time with a message naming the missing engine.

For every question ai_eval builds the workflow inputs from the target's fixed inputs, then the question's own inputs (see below), then puts the question text on the configured input port, and asks the runner to execute the workflow once, synchronously. The runner reports the output text, a normalized status, the steps it ran (as a tree by execution id, so loops and sub-workflows keep their shape), and the model calls it observed.

What that means for a result row:

The response text is graded like any other response, so every grader works unchanged.
A workflow block is stored with each question: engine, run id, status, failure kind, engine seconds, up to 200 model calls, and up to 200 steps, with every text capped at 500 characters and the whole block at 64 KB. When the engine reported more, steps_total and calls_total say how many and truncated is true; when a text was shortened, previews_capped is true. The block is stored on failed questions too, so a pipeline that failed halfway keeps the steps it ran. Expand Workflow run in each question on the results page to see the summary, model calls, nested steps, and available previews. Notes indicate when the stored record or its text was shortened.
token_usage is the run total across all model calls, stored on failed questions as well so the tokens a failed run paid for are still counted. It also carries cached when every call reports a cached count. model_id is set only when every call used the same model; a pipeline that calls two models reports no single model id rather than a misleading one.
A run the engine reports as failed, cancelled, timed out or paused fails the question. A run that completed without delivering any output fails it too. A launch the runner refused before doing any work (unknown workflow, an asynchronous orchestrator it cannot wait on, a missing permission) is stored as failure kind refused, distinct from a run that started and failed.
elapsed stays ai_eval's own wall clock; the engine's own duration is stored separately as engine_seconds inside the block.

The workflow runs as the question's caller (caller_uid, or --as-user from Drush), as agent mode does, and as whoever started the run otherwise. Runs can take as long as the engine allows a pipeline to take, so run workflow targets from Drush or a queue, not from a web request.

Multi-turn questions reach the workflow as the flattened transcript on the one input port, the same degradation agent mode applies.

Create workflow targets in configuration. The add form offers agent and chat only; create a workflow target as config YAML and import it, or set the keys with drush config:set.

The edit form keeps an existing target in workflow mode and shows its engine, workflow ID, question port, fixed inputs, and output path as read-only settings. Saving preserves all workflow configuration while allowing changes to the label, dataset, graders, and other evaluation settings. Change workflow settings in YAML for now. The guided editor links workflow targets to the edit form.

A minimal target:

# config/sync/ai_eval.target.summarize_pipeline.yml
id: summarize_pipeline
label: 'Summarize pipeline'
status: true
mode: workflow
workflow_engine: flowdrop
workflow_id: summarize_v2
workflow_input_port: url
workflow_inputs:
  language: en
  max_words: 200
workflow_output_path: ''
workflow_snapshot: ''
dataset_source: file
dataset_ref: corpus_pages.yaml
graders:
  - format_grader
  - fact_match_grader
threshold: 3.5
metric: avg_score
gate: hard

Saving a workflow target refuses a fixed input on the same port as workflow_input_port, because the question is set on that port after the fixed inputs and a fixed value there could never reach the engine. The runner checks the same rule again at run time for config written with drush config:set.

Per-question inputs¶

A question can carry its own workflow inputs under metadata.inputs, a map of input port name to value:

questions:
  - id: hours_greek
    input: "When does the library open on Saturdays?"
    metadata:
      inputs:
        language: el
        max_words: 50

They are applied over the target's fixed workflow_inputs: on the same port the question's value wins, a new port is added, and values keep their types (50 reaches the engine as a number). A question input on the question port is refused, like a fixed one. Questions without metadata.inputs get the fixed inputs alone, and one question's inputs never carry over to the next. Agent and chat targets ignore metadata.inputs.

When the workflow declares its input ports, the question port and every fixed or question input must name one of them; anything else refuses the launch with a message listing the declared ports. A workflow that declares no ports takes any port name as given. A refused launch fails the question with failure kind refused and the engine is never called.

AI Answers¶

The ai_eval_ai_answers submodule adds a runner for AI Answers, so the answers visitors get on the search page can be graded. The engine is ai_answers, the workflow is an agent enabled for AI Answers, and the question port is question:

mode: workflow
workflow_engine: ai_answers
workflow_id: content_search_agent
workflow_input_port: question

The graded response is the answer text the visitor reads. references: list or rendered (which also quotes each reference card) appends the sources as a Sources: block, which judges then grade as part of the answer; use it only for groundedness, without format criteria. The tool record holds only the tools the agent chose to run; AI Answers' own search is a retrieve step, completed only when it was seen to return references. An answer that found no sources is graded too, with workflow.engine_status set to no_answer. Run it with --as-user=0 to grade what anonymous visitors get. The submodule README has the full target, the options, and the limits. The agent's model calls are recorded with their tokens; judge calls are not.

Search API (retrieval)¶

Enable ai_eval_search_api to grade the final ranked results from an index. The engine is search_api, the workflow id is the index machine name, and the question port must be keys. Create the target as config:

# config/sync/ai_eval.target.retrieval_content.yml
id: retrieval_content
label: 'Retrieval: content_vector'
status: true
mode: workflow
workflow_engine: search_api
workflow_id: content_vector
workflow_input_port: keys
workflow_inputs:
  k: 5
  chunks: true
  min_score: 0.5
workflow_output_path: ''
workflow_snapshot: ''
dataset_source: file
dataset_ref: retrieval_questions.yaml
graders:
  - recall_at_k_grader
  - reciprocal_rank_grader
threshold: 3.5
metric: avg_score
gate: hard

Input	Allowed values	Default
`keys`	Nonempty question text; use as the question port	Required
`k`	Integer from 1 to 50	10
`conjunction`	`AND`, `OR`	`AND`
`chunks`	Boolean; ignored by backends other than AI Search	`true`
`min_score`	Number or null; filters after the query	null
`text`	`snippet`, `none`	`snippet`

Question metadata.inputs overrides fixed options. Expected pages go under expected.items, for example [node/108, entity:node/41:en]. Use caller_uid or --as-user=0 to select the caller. Avoid user_name, which prefixes the keys. Access depends on the backend and index processors. Pages the caller cannot view count as misses. The runner does not bypass access.

Behavior	Database indexes	AI Search indexes
Query	Parsed terms; natural-language questions may need `OR`	Keys are embedded; conjunction has little effect
Results	Index items	Chunks by default; page ids come from backend metadata
Text	Excerpts when the Highlight processor provides them	Chunk content
Access	Index processors, such as `content_access`	Backend checks each item as the current user
Cost	No model call	Embedding provider cost
Stability	Usually stable for fixed indexed data	Approximate search and embeddings may vary

One search step records requested inputs and final observed results. The response preserves returned order, with labels and capped snippets. The runner loads labels through datasources so missing entities do not cause it to delete stale index entries. Site result processors may still do so independently. Aborted queries, empty languages and swallowed AI Search errors fail before any grader runs. A successful empty result scores zero. See the bridge README for the complete response, side effects and limits.

Field automators¶

Enable ai_eval_ai_automators to grade an enabled AI Automators configuration on an unsaved content entity. Use engine ai_automators, the automator config ID as workflow_id, and a source field as the question port:

mode: workflow
workflow_engine: ai_automators
workflow_id: node.article.field_summary.default
workflow_input_port: body
workflow_inputs:
  text_format: plain_text

Set other source fields through workflow_inputs or metadata.inputs. Formatted plain text needs text_format. The target field and identity keys cannot be inputs. Disabled automators, known writing rules and image styles are refused, including changes made by config subscribers. Only one rule runs; dependent field values must be inputs.

The response is the final stored field value in memory. Taxonomy rules with auto-create instead return predicted label lines without constructing or saving terms. This labels-only prediction covers assignment before term lifecycle hooks; renaming hooks and save failures are outside its scope. engine_status distinguishes written, unchanged, empty, and not_stored. The last means taxonomy verification left values, even if assignment would clear a default and return an empty response. A diagnostic labels step classifies existing, new and dropped values; its preview is capped.

The subject is guarded against saves from before construction, and a separate term write barrier covers construction through rendering for every rule. Other entities, site-hook reactions, provider calls and logging remain real; the guards cover the subject entity and taxonomy term saves only. On drupal/ai 1.5.0, bare [] taxonomy replies fail, including replies to similar-tag searches. See the submodule README for setup, supported rules, telemetry, and limits.

AI Prompt¶

Enable ai_eval_ai_prompt to grade stored drupal/ai prompts. Select engine ai_prompt, a prompt id such as ai_ckeditor_summarize__default, and question port inputText. Set the target's provider_id and model_id explicitly. Prompts do not store a model.

Workflow runners receive the question port and target chat settings: provider_id, model_id, system_prompt, max_tokens and temperature. Engines that select their own models may ignore them. Temperature zero remains an explicit override. An empty response limit uses the provider default.

The prompt must contain the question port's own {inputText} placeholder. Set other declared variables through workflow_inputs or metadata.inputs. Double-brace placeholders are not filled. Twig blocks and required Drupal tokens refuse before calling the provider. This evaluates the stored prompt, without the consuming module's wrapping instructions.

The record contains a render step and a generate step. The render preview shows renderer output; subscribers may change what is sent. Model calls are recorded only when the provider was observed returning. Typed rate-limit refusals may retry before a response, but quota errors and post-response failures never do. See the bridge README for snapshot semantics, recorded evidence and limits.

All three modes share the same datasets, graders, scoring, gates, and reporting.

Creating a target¶

Go to /admin/config/ai/ai-eval.
Use the Add eval target action (/admin/config/ai/ai-eval/targets/add).
Fill in the fields below and save.

Clicking a target's name afterwards opens its workflow hub (/admin/config/ai/ai-eval/targets/<id>/workflow), which recommends the next step depending on whether the target still needs cases, needs error analysis, or is ready to operate and optimize.

The error analysis steps count this test's work: the annotated questions on its results (one per question, however many reviewers annotated it) and the failure modes those annotations use. A failure mode that no annotation of this test uses does not mark grouping done. Imported conversations are not linked to a test, so Review conversations counts reviews across the whole site and says so, and the Annotate step names them without counting them. The workflow page's Reviewed conversations (whole site) rates are the site's as well.

Steps that work on part of this test open that part: Author cases opens its dataset when the dataset is on this site, Define rubric the rubrics its questions name, and Write & validate judge the validation page of each LLM judge it uses. With several rubrics or judges the action is a dropbutton, and with none the step opens the full list. The workflow does not offer the optimizer to a chat or workflow test, which only rewrites an agent's prompt.

Field reference¶

Field	Stored as	Notes
Label	`label`	Required. Display name.
Machine name	`id`	Required, fixed after the first save.
Evaluation mode	`mode`	`agent`, `chat` or `workflow`. Decides which of the mode sections below applies. The add form offers agent and chat. Existing workflow targets keep their mode and show read-only workflow settings; change those in YAML.
Agent ID	`agent_id`	Agent mode only. A select of installed `ai_agents` plugins, or a text field when `ai_agents` is absent.
Provider / Model	`provider_id`, `model_id`	Editable in chat mode. AI Prompt workflow targets set these in YAML; the form hides chat settings for workflow targets. A select of the AI module's chat provider and model pairs. If the provider list cannot be loaded it falls back to a text field expecting `provider__model`, for example `openai__gpt-4o-mini`.
System prompt	`system_prompt`	Editable in chat mode; set in YAML for AI Prompt workflow targets. Prepended to every question. Ignored in agent mode.
Max response tokens	`max_tokens`	Editable in chat mode; set in YAML for AI Prompt workflow targets. Empty means the provider default. A modest cap lowers cost and avoids upfront affordability errors on metered providers.
Sampling temperature	`temperature`	Editable in chat mode; set in YAML for AI Prompt workflow targets, 0 to 2. Empty means the provider default, which is normally what you want: the target is the system under test and should run at its production temperature.
Dataset	`dataset_source`, `dataset_ref`	Required. One grouped select, with an optgroup per source (File, Config, Entity). The chosen option is split into the source plugin id and the reference within it.
Graders	`graders`	Checkboxes in two groups, LLM judges and deterministic checks. Each label carries the scale that grader answers on. Every selected grader scores every question.
Must pass	`must_pass`	Quality Gate. One checkbox per selected grader. A score below the question pass threshold fails the question and the gate under either metric. A grader that does not apply adds no failure.
Question pass threshold	`question_pass_threshold`	0 to 5. Empty uses the global default from Settings.
Response char limit	`response_char_limit`	Minimum 100. Caps the response length `format_grader` scores against. Empty uses the global default. LLM judges are not bounded by it; they are bounded separately by `judge_response_char_limit`.
Gate threshold	`threshold`	Required. Measured on the metric's own scale: 0 to 5 for `avg_score`, 0 to 1 for `pass_rate`.
Gate metric	`metric`	`avg_score` (Average score) or `pass_rate` (Pass rate).
Gate type	`gate`	A hard miss makes `drush ai-eval:run` exit `1`. A soft miss is named in the output and does not change its exit code by itself. Both show the same Pass or Fail on screens.
Proving-ground fixtures	`fixtures`	Optional, one `name@version` pin per line, for example `kb_docs@1`. Names the fixtures an eval bed provisions and resets for this target. Leave empty when the target does not run against a proving-ground environment.
Workflow engine	`workflow_engine`	Workflow mode only, YAML for now. The id of a registered workflow runner, for example `flowdrop`.
Workflow	`workflow_id`	Workflow mode only. The workflow's id within that engine.
Question input port	`workflow_input_port`	Workflow mode only. The declared input that receives the question text.
Fixed inputs	`workflow_inputs`	Workflow mode only. A map of port name to value sent with every question. Values keep their type (string, number, boolean, list, map). Must not contain the question port.
Output path	`workflow_output_path`	Workflow mode only, optional. A dotted path into the declared pipeline output; empty reads the declared output as a whole.
Workflow snapshot	`workflow_snapshot`	Workflow mode only, optional. A hash or version of the workflow definition at save time, stored with every result so a later edit to the workflow is visible.

The gate threshold field carries a live note reading / 5 ceiling. That ceiling is the shared composite scale, not the scale of any one grader: every grader is mapped onto 0 to 5 before scores are averaged, so a target graded only by a binary judge is still gated on 0 to 5, and 3.5 is a sensible threshold for it.

What the form refuses¶

A must-pass box for an unselected grader is ignored, not refused. Only selected graders can be saved as must pass. Existing targets default to an empty list.

Four checks run on save.

A pass_rate threshold above 1.0 is rejected. A pass rate cannot exceed 100 percent, and the avg_score default of 4.0 left behind after a metric switch is the usual way to hit this.
An avg_score threshold above 5.0 is rejected as an unpassable gate. No run could ever reach it.
A malformed or duplicated fixture pin is rejected, so a corrupt binding never reaches the entity.
A manually typed provider and model without the __ separator is rejected.

One case produces a warning rather than an error: an avg_score threshold between 0 and 1 saves, but the form points out that such a threshold passes almost any run and asks whether you meant pass_rate.

Reading the Targets list¶

The landing page is a portfolio view. Each row carries the gate, the latest average score against its threshold, the pass rate, the trust state of the judges that target uses, when it last ran, how many cases it holds, and a Run button.

Opening a Run URL directly takes you to the target's workflow page without starting a test. Use the Run button there to start it.

Two badges can appear next to the trust state:

Text only, on an agent-mode target with no grader that can observe what the agent did.
Judge same family, when the configured judge shares a coarse model family with the target's own model. Models prefer their own family's output, so part of every score is the judge grading its own reflection. This marker is row-level only for chat-mode targets, because an agent-mode target exposes no model id in config; for those the comparison is reported per run instead. See trusting your judges.

Neither badge blocks anything. Both qualify what the numbers next to them mean.

Changing the gate metric updates an empty threshold, the previous default, or a value above the new scale. A custom value that fits the new scale stays.