The prompt optimizer

When a target sits below its gate, the optimizer reads that run's failures, asks an LLM for a better agent system prompt, evaluates the candidate on the same dataset, and records it as applied, pending, or rejected. This page is for whoever runs it and reviews what it produces.

The optimizer rewrites agent system prompts, so chat-mode targets are skipped by design. See targets for the difference.

flowchart TD
    B["Baseline: run the target N times"]
    G{"Above the gate?"}
    SKIP["Skip this target"]
    P["Proposer model writes a candidate prompt"]
    C["Candidate: run the target N times with it"]
    D{"Candidate beats baseline by more than the margin?"}
    R["Rejected, prompt reverted"]
    RV{"Review required?"}
    PEND["Stored pending for a human decision"]
    APP["Applied to the agent config"]

    B --> G
    G -->|"yes"| SKIP
    G -->|"no"| P --> C --> D
    D -->|"no"| R
    D -->|"yes"| RV
    RV -->|"yes, the default"| PEND
    RV -->|"no"| APP

Running it

# Improve prompts where a target sits below its threshold
drush ai-eval:optimize --target=my_target

# Store proposals for review instead of applying them
drush ai-eval:optimize --target=my_target --propose

ai-eval:optimize (alias aeo) takes:

Option	Default	What it does
--target	all enabled targets	Optimize one target by machine name.
--max-attempts	max_optimization_attempts, 3	How many candidates to try per target.
--propose	off	Store every candidate as pending instead of applying it.
--split	train	Which partition to iterate on. train improves on the train questions and reports against validation. validation and all both iterate and score on that one set. test is refused.
--allow-same-family	off	Proceed when the proposer and the judge share a model family. Affected candidates are flagged spb_risk.
--strict-resolution	off	Refuse candidates whose gain no judge in the run can detect, instead of applying them with a warning.
--dry-run	off	Report how many failures a candidate would be built from, then stop.

Warning

--dry-run is not free. The baseline runs before the dry-run check, so the command still executes the target optimizer_candidate_runs times against real providers. What it skips is the proposal and the candidate evaluation.

The command reports per target on stdout and stores every candidate, including rejected ones, in the optimization table. It does not set a gate-style exit code: use ai-eval:run for that.

The proposer is not the judge

Two models are involved, and they must not be the same family. The judge scores the candidate's output; the proposer writes the candidate prompt. A judge over-scores prompts written by its own model family, which contaminates exactly the number the accept decision rests on.

Configure the proposer with proposer_provider and proposer_model in configuration. The resolution rule:

1. Both proposer settings set: that provider and model propose.
2. Either setting empty: the proposer falls back to the site judge provider and model (judge_provider, judge_model, never a per-judge override), and the command warns once per run that you should configure a distinct proposer.
3. The proposer is compared with every LLM judge on the target, each on the model it calls (its own override, else the site judge). A configured proposer that shares a family with any of them: the command refuses to optimize that target and tells you to pick another family, unless you pass --allow-same-family, which downgrades the refusal to a warning and flags every candidate produced under it with spb_risk.

Family comparison is a coarse prefix heuristic for OpenAI, Anthropic and Google models. Aggregators such as OpenRouter are supported when their model id carries the vendor namespace, for example anthropic/claude-sonnet-4. An identity the heuristic cannot resolve is reported as unknown and flagged as a self-preference risk rather than being treated as verified different.

Why every measurement is repeated

An LLM judge is noisy. Run the same prompt twice and the score moves, and that movement is indistinguishable from an improvement if you only measure once.

So both sides are repeated. The baseline is run optimizer_candidate_runs times (3 by default) and averaged, and each candidate is run the same number of times and averaged. Comparing an N-run candidate against a single-run baseline would average the noise out of only one side and make the comparison wobble, so the baseline pays the same cost.

Under the held-out protocol the baseline runs twice over: once on the train split, which gives the proposer its failure picture, and once on the validation split, which produces the number every candidate is compared against.

Raising optimizer_candidate_runs buys a steadier comparison and costs linearly more provider calls. Lowering it to 1 turns the comparison back into a coin flip.

The minimum dataset size

A target whose dataset is smaller than optimizer_min_dataset_size (15 questions by default) is skipped with a warning before any baseline tokens are spent. On a small dataset the per-run score variance routinely exceeds the improvement margin, so the optimizer would be tuning prompts against judge noise.

Once splits are active the guard counts the partition being iterated on, not the whole file: 14 train questions out of a 20-row dataset are still 14 questions of signal.

The improvement margin

A candidate is accepted only when its mean score beats the current baseline by more than improvement_margin, 0.4 points on the shared 0 to 5 scale by default. Everything else is rejected, with the reason named:

Outcome	Condition
REGRESSION	The candidate scored below the baseline. Never queued for review, even in propose mode.
TIE	Equal score. Nothing for a reviewer to act on.
REVERTED	Better, but by less than the margin.
PENDING	Cleared the margin, and review is required.
ACCEPTED	Cleared the margin, and review is not required.

The margin exists to sit above judge noise, and a validated judge can tell you where that noise floor is. Validation on a fixture with ranked pairs records a judge's discrimination resolution: the smallest true quality gap it ranks reliably. See judges.

The optimizer reads those recorded resolutions per judge and uses them two ways. Before iterating, it prints each judge's floor and warns when the floor is above your margin, because a margin below every judge's resolution accepts candidates no judge in the run can vouch for. After a candidate is scored, it checks whether any single answer improved by at least the resolution of the judge that scored it, counting only improvements that hold on every run. If none did, it warns that the aggregate gain is not evidence of a better prompt, whatever its sign. --strict-resolution turns that warning into a refusal and stops iterating the target. The check is per judge and per answer, never an aggregate gain against a floor, because a composite average mixes judges and questions and bounds neither.

When no judge in the run has a recorded resolution, or no question was comparable on both sides, the check is reported as skipped rather than passed.

Proposals need review by default

require_review in ai_eval.settings ships as true, so out of the box no candidate is permanently applied to your agent config without a human. Every candidate that clears the margin is stored as pending and waits for review. --propose forces the same behavior for one invocation regardless of the setting.

Warning

Review governs the final write, not every write. Evaluating a candidate means saving it to the live agent config, running the dataset against it, and restoring the original in a finally block. That happens in review mode too. For the length of each candidate evaluation the agent really is running the candidate prompt, so anything else hitting that agent at the same time sees it. Do not run the optimizer against an agent serving production traffic.

Pending candidates queue at /admin/config/ai/ai-eval/optimize. The list and detail pages need operate ai eval or administer ai eval; applying or rejecting a candidate needs administer ai eval.

Setting require_review to false lets a cleared candidate be written straight to the agent's system_prompt. That is a deliberate choice to let an automated process edit a production prompt with no human in the path.

Safety around a live agent

Candidate evaluation clones the config agent's backing entity in memory and sets the candidate prompt on that agent instance only. It does not save the candidate to shared agent configuration. Concurrent invocations continue to read the live prompt, and an interrupted evaluation leaves no temporary candidate prompt to restore.

Only the existing explicit apply path (or accepted automatic application when require_review is false) writes a candidate to configuration. Agents that do not support an in-memory config-entity override fail candidate invocation instead of silently evaluating the original prompt.

An accepted candidate becomes the baseline the next attempt is measured against, so attempt 3 is compared against the prompt actually in force rather than against the original.

Splits and the sealed test set

With no declared split, the optimizer improves the prompt on the same questions it scores against, which inflates the reported gain. It does this with one loud warning per run rather than refusing, since it is the historic behavior. Label questions with split: or add a splits: block to hold out a validation set; see datasets.

--split=test is refused outright. The test partition stays sealed for a final unbiased reading, and for any split other than train the optimizer both iterates on and scores against that same set, which would burn the held-out questions it exists to protect. Only train splits the two roles, improving on train and reporting against validation. Measure against test with drush ai-eval:run --split=test, which only reads.

Related commands

drush ai-eval:distill is the optimizer's quieter sibling. It writes markdown summaries of stored results, useful as compact context for further analysis, and changes nothing.