The prompt optimizer

When a target sits below its gate, the optimizer reads that run's failures, asks an LLM for a better agent system prompt, evaluates the candidate on the same dataset, and records it as applied, pending, or rejected. This page is for whoever runs it and reviews what it produces.

The optimizer rewrites agent system prompts, so chat-mode targets are skipped by design. See targets for the difference.

flowchart TD
    B["Baseline: run the target N times"]
    G{"Above the gate?"}
    SKIP["Skip this target"]
    P["Proposer model writes a candidate prompt"]
    C["Candidate: run the target N times with it"]
    D{"Candidate beats baseline by more than the margin?"}
    R["Rejected, prompt reverted"]
    RV{"Review required?"}
    PEND["Stored pending for a human decision"]
    APP["Applied to the agent config"]

    B --> G
    G -->|"yes"| SKIP
    G -->|"no"| P --> C --> D
    D -->|"no"| R
    D -->|"yes"| RV
    RV -->|"yes, the default"| PEND
    RV -->|"no"| APP

Running it

# Improve prompts where a target sits below its threshold
drush ai-eval:optimize --target=my_target

# Store proposals for review instead of applying them
drush ai-eval:optimize --target=my_target --propose

ai-eval:optimize (alias aeo) takes:

Option	Default	What it does
--target	all enabled targets	Optimize only this target.
--max-attempts	max_optimization_attempts, 3	How many candidates to try per target.
--propose	off	Store every candidate as pending instead of applying it.
--split	train	Which partition to iterate on. train improves on the train questions and reports against validation. validation and all both iterate and score on that one set. test is refused.
--allow-same-family	off	Proceed when the proposer and the judge share a model family. Affected candidates are flagged spb_risk.
--strict-resolution	off	Refuse candidates whose gain no judge in the run can detect, instead of applying them with a warning.
--dry-run	off	Report how many failures a candidate would be built from, then stop.

Warning

--dry-run is not free. The baseline runs before the dry-run check, so the command still executes the target optimizer_candidate_runs times against real providers. What it skips is the proposal and the candidate evaluation.

The command reports per target on stdout and stores every candidate, including rejected ones, in the optimization table. It does not set a gate-style exit code: use ai-eval:run for that.

The proposer is not the judge

Two models are involved, and they must not be the same family. The judge scores the candidate's output; the proposer writes the candidate prompt. A judge over-scores prompts written by its own model family, which contaminates exactly the number the accept decision rests on.

Configure the proposer with proposer_provider and proposer_model in configuration. The resolution rule:

1. Both proposer settings set: that provider and model propose.
2. Either setting empty: the proposer falls back to the judge provider and model, and the command warns once per run that you should configure a distinct proposer.
3. A proposer is configured but resolves to the judge's model family: the command refuses to run and tells you to pick another family, unless you pass --allow-same-family, which downgrades the refusal to a warning and flags every candidate produced under it with spb_risk.

Family comparison is a coarse prefix heuristic, and it only over-groups inside the three provider ids it knows: openai, anthropic and google. For any other provider id, including an aggregator such as OpenRouter, the table has no entry and every model becomes its own family. Two models that really are siblings then compare as different, the refusal never fires, and no spb_risk flag is set. Check the pairing yourself when your provider is not one of those three.

Why every measurement is repeated

An LLM judge is noisy. Run the same prompt twice and the score moves, and that movement is indistinguishable from an improvement if you only measure once.

So both sides are repeated. The baseline is run optimizer_candidate_runs times (3 by default) and averaged, and each candidate is run the same number of times and averaged. Comparing an N-run candidate against a single-run baseline would average the noise out of only one side and make the comparison wobble, so the baseline pays the same cost.

Under the held-out protocol the baseline runs twice over: once on the train split, which gives the proposer its failure picture, and once on the validation split, which produces the number every candidate is compared against.

Raising optimizer_candidate_runs buys a steadier comparison and costs linearly more provider calls. Lowering it to 1 turns the comparison back into a coin flip.

The minimum dataset size

A target whose dataset is smaller than optimizer_min_dataset_size (15 questions by default) is skipped with a warning before any baseline tokens are spent. On a small dataset the per-run score variance routinely exceeds the improvement margin, so the optimizer would be tuning prompts against judge noise.

Once splits are active the guard counts the partition being iterated on, not the whole file: 14 train questions out of a 20-row dataset are still 14 questions of signal.

The improvement margin

A candidate is accepted only when its mean score beats the current baseline by more than improvement_margin, 0.4 points on the shared 0 to 5 scale by default. Everything else is rejected, with the reason named:

Outcome	Condition
REGRESSION	The candidate scored below the baseline. Never queued for review, even in propose mode.
TIE	Equal score. Nothing for a reviewer to act on.
REVERTED	Better, but by less than the margin.
PENDING	Cleared the margin, and review is required.
ACCEPTED	Cleared the margin, and review is not required.

The margin exists to sit above judge noise, and a validated judge can tell you where that noise floor is. Validation on a fixture with ranked pairs records a judge's discrimination resolution: the smallest true quality gap it ranks reliably. See judges.

The optimizer reads those recorded resolutions per judge and uses them two ways. Before iterating, it prints each judge's floor and warns when the floor is above your margin, because a margin below every judge's resolution accepts candidates no judge in the run can vouch for. After a candidate is scored, it checks whether any single answer improved by at least the resolution of the judge that scored it, counting only improvements that hold on every run. If none did, it warns that the aggregate gain is not evidence of a better prompt, whatever its sign. --strict-resolution turns that warning into a refusal and stops iterating the target. The check is per judge and per answer, never an aggregate gain against a floor, because a composite average mixes judges and questions and bounds neither.

When no judge in the run has a recorded resolution, or no question was comparable on both sides, the check is reported as skipped rather than passed.

Proposals need review by default

require_review in ai_eval.settings ships as true, so out of the box no candidate is permanently applied to your agent config without a human. Every candidate that clears the margin is stored as pending and waits for review. --propose forces the same behavior for one invocation regardless of the setting.

Warning

Review governs the final write, not every write. Evaluating a candidate means saving it to the live agent config, running the dataset against it, and restoring the original in a finally block. That happens in review mode too. For the length of each candidate evaluation the agent really is running the candidate prompt, so anything else hitting that agent at the same time sees it. Do not run the optimizer against an agent serving production traffic.

Pending candidates queue at /admin/config/ai/ai-eval/optimize. The list and detail pages need operate ai eval or administer ai eval; applying or rejecting a candidate needs administer ai eval.

Setting require_review to false lets a cleared candidate be written straight to the agent's system_prompt. That is a deliberate choice to let an automated process edit a production prompt with no human in the path.

Safety around a live agent

Evaluating a candidate means temporarily writing it to the agent config. Before that write the optimizer registers a PHP shutdown function that restores the original prompt, and the normal path restores it in a finally block.

Between them these cover an exception, a fatal error, and an ordinary non-zero exit. They do not cover everything:

Interruption	Original prompt restored
Exception or fatal error	Yes, by the shutdown function
Ordinary non-zero exit	Yes
SIGTERM (plain kill, docker stop, systemd stop)	No. PHP does not run shutdown functions on SIGTERM
SIGKILL (kill -9)	No, for the same reason
Power loss, OOM killer, container kill	No
SIGINT (Ctrl-C)	Not reliably. The module installs no signal handler, so the outcome depends on what the process was doing

The module registers no signal handler anywhere, so nothing converts a signal into the clean shutdown the guard needs.

Warning

After any interrupted optimizer run, not only a kill -9, check the agent's system_prompt before trusting it. It may still hold an unreviewed candidate. The log line the shutdown handler writes on an unclean exit is your signal that the guard did fire; its absence after an interruption means it did not.

An accepted candidate becomes the baseline the next attempt is measured against, so attempt 3 is compared against the prompt actually in force rather than against the original.

Splits and the sealed test set

With no declared split, the optimizer improves the prompt on the same questions it scores against, which inflates the reported gain. It does this with one loud warning per run rather than refusing, since it is the historic behavior. Label questions with split: or add a splits: block to hold out a validation set; see datasets.

--split=test is refused outright. The test partition stays sealed for a final unbiased reading, and for any split other than train the optimizer both iterates on and scores against that same set, which would burn the held-out questions it exists to protect. Only train splits the two roles, improving on train and reporting against validation. Measure against test with drush ai-eval:run --split=test, which only reads.

Related commands

drush ai-eval:distill is the optimizer's quieter sibling. It writes markdown summaries of stored results, useful as compact context for further analysis, and changes nothing.