The prompt optimizer¶

When a target sits below its gate, the optimizer reads that run's failures, asks an LLM for a better agent system prompt, evaluates the candidate on the same dataset, and records it as applied, pending, or rejected. This page is for whoever runs it and reviews what it produces.

The optimizer rewrites agent system prompts, so chat and workflow targets are skipped before any model call. Agent tests without an agent system prompt are also skipped before the baseline. The workflow page matches this: for these tests it shows no Optimize prompt link. For an agent without a system prompt it says "No system prompt" and tells you to add one to the agent, then run the test again. See targets for the difference.

flowchart TD
    B["Baseline: run the target N times"]
    G{"Above the gate?"}
    SKIP["Skip this target"]
    P["Proposer model writes a candidate prompt"]
    C["Candidate: run the target N times with it"]
    D{"Candidate beats baseline by more than the margin?"}
    R["Rejected, live prompt unchanged"]
    RV{"Review required?"}
    PEND["Stored pending for a human decision"]
    APP["Applied to the agent config"]

    B --> G
    G -->|"yes"| SKIP
    G -->|"no"| P --> C --> D
    D -->|"no"| R
    D -->|"yes"| RV
    RV -->|"yes, the default"| PEND
    RV -->|"no"| APP

Running it¶

# Improve prompts where a target sits below its threshold
drush ai-eval:optimize --target=my_target

# Store proposals for review instead of applying them
drush ai-eval:optimize --target=my_target --propose

ai-eval:optimize (alias aeo) takes:

Option	Default	What it does
`--target`	all enabled targets	Optimize one target by machine name.
`--max-attempts`	`max_optimization_attempts`, 3	How many candidates to try per target.
`--propose`	off	Store every candidate as pending instead of applying it.
`--split`	`train`	Which partition to iterate on. `train` improves on the train questions and reports against validation. `validation` and `all` both iterate and score on that one set. `test` is refused.
`--allow-same-family`	off	Proceed when the proposer and the judge share a model family. Affected candidates are flagged `spb_risk`.
`--strict-resolution`	off	Refuse candidates whose gain no judge in the run can detect, instead of applying them with a warning.
`--dry-run`	off	Report how many failures a candidate would be built from, then stop.

Warning

--dry-run is not free. The baseline runs before the dry-run check, so the command still executes the target optimizer_candidate_runs times against real providers. What it skips is the proposal and the candidate evaluation.

The command reports per target on stdout and stores each compared candidate, including rejected ones, in the optimization table. Attempts with insufficient coverage are reported and not stored. It does not set a gate-style exit code: use ai-eval:run for that.

The proposer is not the judge¶

Two models are involved, and they must not be the same family. The judge scores the candidate's output; the proposer writes the candidate prompt. A judge over-scores prompts written by its own model family, which contaminates exactly the number the accept decision rests on.

Configure the proposer with proposer_provider and proposer_model in configuration. The resolution rule:

Both proposer settings set: that provider and model propose.
Either setting empty: the proposer falls back to the site judge provider and model (judge_provider, judge_model, never a per-judge override), and the command warns once per run that you should configure a distinct proposer.
The proposer is compared with every LLM judge on the target, each on the model it calls (its own override, else the site judge). A configured proposer that shares a family with any of them: the command refuses to optimize that target and tells you to pick another family, unless you pass --allow-same-family, which downgrades the refusal to a warning and flags every candidate produced under it with spb_risk.

Family comparison is a coarse prefix heuristic for OpenAI, Anthropic and Google models. Aggregators such as OpenRouter are supported when their model id carries the vendor namespace, for example anthropic/claude-sonnet-4. An identity the heuristic cannot resolve is reported as unknown and flagged as a self-preference risk rather than being treated as verified different.

Why every measurement is repeated¶

An LLM judge is noisy. Run the same prompt twice and the score moves, and that movement is indistinguishable from an improvement if you only measure once.

So both sides are repeated. The baseline is run optimizer_candidate_runs times (3 by default) and averaged, and each candidate is run the same number of times and averaged. Comparing an N-run candidate against a single-run baseline would average the noise out of only one side and make the comparison wobble, so the baseline pays the same cost.

Under the held-out protocol the baseline runs twice over: once on the train split, which gives the proposer its failure picture, and once on the validation split, which produces the number every candidate is compared against.

Raising optimizer_candidate_runs buys a steadier comparison and costs linearly more provider calls. Lowering it to 1 turns the comparison back into a coin flip.

The minimum dataset size¶

A target whose dataset is smaller than optimizer_min_dataset_size (15 questions by default) is skipped with a warning before any baseline tokens are spent. On a small dataset the per-run score variance routinely exceeds the improvement margin, so the optimizer would be tuning prompts against judge noise.

Once splits are active the guard counts the partition being iterated on, not the whole file: 14 train questions out of a 20-row dataset are still 14 questions of signal.

A baseline already above its threshold is skipped when at least half of the questions were measured.

Questions that could not be scored¶

A question that errors, for example when the judge fails on a provider error, is left out of a run's average. The optimizer compares the baseline and the candidate only on the questions both were scored on in every repeat, and says how many that was. If fewer than half of a run's questions were scored, the target (for a baseline) or the attempt (for a candidate) is skipped, nothing is stored for the attempt, and the output says which run. Fix the errors and run again. A question no grader can score, for example one with no expected facts, is not an error, but it is also excluded from measured coverage. The Optimizations page repeats this rule.

The improvement margin¶

A candidate is accepted only when its mean score beats the current baseline by more than improvement_margin, 0.4 points on the shared 0 to 5 scale by default. Everything else is rejected, with the reason named:

Outcome	Condition
`REFUSED`	With `--strict-resolution`, no answer improved by an amount its judge can detect. Stored as rejected, and iteration stops.
`REGRESSION`	The candidate scored below the baseline. Never queued for review, even in propose mode.
`TIE`	Equal score. Nothing for a reviewer to act on.
`REVERTED`	Better, but by less than the margin.
`PENDING`	Cleared the margin, and review is required.
`ACCEPTED`	Cleared the margin, and review is not required.

The margin exists to sit above judge noise, and a validated judge can tell you where that noise floor is. Validation on a fixture with ranked pairs records a judge's discrimination resolution: the smallest true quality gap it ranks reliably. See judges.

The optimizer reads those recorded resolutions per judge and uses them two ways. Before iterating, it prints each judge's floor and warns when the floor is above your margin, because a margin below every judge's resolution accepts candidates no judge in the run can vouch for. After a candidate is scored, it checks whether any single answer improved by at least the resolution of the judge that scored it, counting only improvements that hold on every run. If none did, it warns that the aggregate gain is not evidence of a better prompt, whatever its sign. --strict-resolution turns that warning into a refusal and stops iterating the target. The check is per judge and per answer, never an aggregate gain against a floor, because a composite average mixes judges and questions and bounds neither.

When no judge in the run has a recorded resolution, or no question was comparable on both sides, the check is reported as skipped rather than passed.

Proposals need review by default¶

require_review in ai_eval.settings ships as true, so out of the box no candidate is applied to your agent config without a human. Every candidate that clears the margin is stored as pending and waits for review. --propose forces the same behavior for one invocation regardless of the setting.

Candidate evaluation uses an in-memory prompt override. It does not save the candidate to live agent config, so concurrent requests keep using the live prompt. Only applying an accepted candidate writes to config; see safety around a live agent.

Pending candidates queue at /admin/config/ai/ai-eval/optimize. The list and detail pages need operate ai eval or administer ai eval; applying or rejecting a candidate needs administer ai eval.

Setting require_review to false lets a cleared candidate be written straight to the agent's system_prompt. That is a deliberate choice to let an automated process edit a production prompt with no human in the path.

Safety around a live agent¶

Candidate evaluation clones the config agent's backing entity in memory and sets the candidate prompt on that agent instance only. It does not save the candidate to shared agent configuration. Concurrent invocations continue to read the live prompt, and an interrupted evaluation leaves no temporary candidate prompt to restore.

Only the existing explicit apply path (or accepted automatic application when require_review is false) writes a candidate to configuration. Agents that do not support an in-memory config-entity override fail candidate invocation instead of silently evaluating the original prompt.

An accepted candidate becomes the baseline the next attempt is measured against, so attempt 3 is compared against the prompt actually in force rather than against the original.

Splits and the sealed test set¶

With no declared split, the optimizer improves the prompt on the same questions it scores against, which inflates the reported gain. It does this with one loud warning per run rather than refusing, since it is the historic behavior. Label questions with split: or add a splits: block to hold out a validation set; see datasets.

--split=test is refused outright. The test partition stays sealed for a final unbiased reading, and for any split other than train the optimizer both iterates on and scores against that same set, which would burn the held-out questions it exists to protect. Only train splits the two roles, improving on train and reporting against validation. Measure against test with drush ai-eval:run --split=test, which only reads.

drush ai-eval:distill is the optimizer's quieter sibling. It writes markdown summaries of stored results, useful as compact context for further analysis, and changes nothing.