Troubleshooting¶

Symptoms you will hit while running evals, what causes each one, and what to do about it. Every entry names the code or the message that produces it, so you can confirm the diagnosis before changing anything.

Most of these show up in one of three places: the ai-eval:run output, the ai_eval logger channel (admin/reports/dblog), or the per-grader reason text on a result at /admin/config/ai/ai-eval/results/{id}. Read the reason text first. Every grader that scored nothing says why it scored nothing.

A run produces no scores¶

The target reports 0.00/5.0 and every question is empty¶

The log carries No questions loaded for target X from Y from the ai_eval channel. The dataset resolved to zero questions, and Scorer::scoreTarget() returns avg_score: 0.0, pass_rate: 0.0, passed_gate: FALSE for an empty question set.

FileSource::load() returns an empty array, without an error, in four cases:

Cause	Fix
The `dataset_ref` contains a directory separator	Use a bare filename. `basename($ref) !== $ref` returns empty by design, so no target can read outside the dataset directory.
The file is not in the configured dataset directory	Check Dataset path at `/admin/config/ai/ai-eval/settings`. Empty config means the module's own `data/` directory.
The YAML failed to parse	A `ParseException` is swallowed and reads as an empty dataset. Run the file through a YAML parser, or validate it against `schema/dataset.schema.json`.
The document has no top-level `questions` key	See datasets for the required shape.

Every row that is not an array is dropped with Skipping malformed dataset row in @dataset in the log, and rows carrying disabled: true are dropped silently.

Every question scores 0.00 and every grader says "skipping"¶

The graders ran and all of them declined. A grader that returns a NULL score without the error flag is a not-applicable skip: it is excluded from the average silently, and when nothing is left the average is 0.0, which is below the default question pass threshold of 3.5, so the question fails.

Check the skip reasons on the result:

no expected_facts; skipping from fact_match_grader: the question carries no expected_facts.
no expected_tools in context, skipping from tool_usage_grader: the question carries no expected_tools.
no executable rubric checks, skipping from rubric_checks: the question has no resolvable rubric_ref, or the rubric's checks all name a kind with no registered executor.

The fix is on the dataset, not the target: attach the assertions those graders need. See rubrics and graders.

The gate FAILs even though the questions that scored look good¶

The result carries insufficient_coverage: TRUE. Scorer fails the gate outright when fewer than half a target's questions could be scored, whatever the survivors' average was. A transient judge outage must not read as a PASS on a lucky subset.

A question is excluded when any of its graders returned an errored result. The ai-eval:run line for the target ends with (N errors) when this happened. Fix the errored grader and re-run; the coverage guard clears itself.

A judge returns no score, or an unusable one¶

LlmJudgeBase::parseJudgeResponse() produces four distinct failure reasons, and all four are errored results, so they flag the question degraded and pull it out of the gate.

Reason text	What happened
`could not parse judge response as JSON`	After stripping code fences and slicing from the first `{` to the last `}`, the text did not decode to an array with a `score` key.
`judge returned a non-numeric score: ...`	The `score` key existed but held something like `"ungrounded"`. Since #3594661 this errors instead of casting to `0.0` and being recorded as a genuine worst-possible score.
`judge answered off its own scale: ...`	The number was outside the range the grader declares. A binary judge asked for 0 or 1 that answers 3 did not answer.
`judge error: ...`	The provider call itself threw. The first 100 characters of the exception are in the reason.

Fixes, in order of how often they work:

Re-run. A single malformed reply is usually transient.
Raise Judge response character limit (judge_response_char_limit, default 4000) if the responses being judged are long. The judge sees the response truncated to that many characters.
Use a different judge model. Small models are the usual source of prose wrapped around the JSON, and of scores outside the requested scale.
If you edited the judge prompt, check that your override still asks for {"score": N, "reason": "brief"}. See judges.

A judge error on every question of every target is not a judge problem. See the next section.

The provider is not configured¶

Status report says "AI Eval judge configuration: Missing"¶

hook_requirements() raises a runtime error when judge_provider or judge_model is empty. Both ship empty in config/install, so a fresh install always shows this until you set them.

Set the judge at /admin/config/ai/ai-eval/settings. When the AI module can list its models you get a select; when it cannot, the form falls back to a text field and warns The list of AI provider models could not be loaded. In that case type the value as provider__model, for example openai__gpt-4o-mini. The form rejects anything without the double underscore.

A chat-mode target scores "invocation failed" on every question¶

EvalRunner::invokeChat() catches everything and logs Chat invocation failed: @msg to the ai_eval channel, then the question is recorded with error: invocation failed. The provider exception is only in the log, so read the log line, not the result. Common contents:

The provider plugin ID on the target does not exist, so createInstance() threw.
The provider has no API key configured, which is a setting on the AI module, not on this module.
The model ID is not one that provider serves.

Confirm the provider works outside ai_eval first, then check the target's provider and model at /admin/config/ai/ai-eval/targets/{id}/edit. See configuration.

The run pauses for long stretches, or dies with a rate-limit exception¶

RateLimitHandler sleeps rate_limit_delay_between_calls seconds (default 2.0) before every provider call, and on an AiRateLimitException or AiQuotaException it retries with exponential backoff: rate_limit_base_delay * 2^(attempt-1), default 5.0 seconds, up to rate_limit_max_retries attempts, default 3. Generic exceptions whose message matches a rate-limit pattern take the same path. After the last attempt the exception is rethrown and the caller records a failure.

Retries are logged as Rate limited (attempt @attempt/@max), retrying in @delay s. If you see those, raise the delay between calls at /admin/config/ai/ai-eval/settings. Some providers want 1000 ms or more between calls; the field is in seconds and accepts fractions.

Agent mode fails¶

"The ai_agents module is required for running evaluations"¶

ai_agents is a Composer suggestion, not a dependency, so the module installs and runs chat mode without it. EvalRunner::invokeAgent() throws this RuntimeException when the agent plugin manager is absent. Install and enable ai_agents, or switch the target to chat mode. See targets.

The log says "Agent @id not found"¶

ai_agents is enabled but no plugin definition matches the target's agent ID. The question is recorded with error: invocation failed. Check the agent ID against the installed agents; a renamed or removed agent config leaves the target pointing at nothing.

The agent ran but the answer is empty¶

Two log lines distinguish the cases:

Agent @id returned solvability @s (not solvable): the agent declared it could not do the job. The runner still calls solve() and answerQuestion() and takes whatever comes back, so an empty result here means both were empty too.
Agent solved but returned empty for @id: the agent reported it could proceed and then produced nothing.

Either way the question is scored as an invocation failure. Reproduce the same input against the agent outside ai_eval before changing the dataset.

A tool assertion fails though the tool ran¶

The verdict ends with "no tool-call record available"¶

tool_usage_grader has two sources of truth. First choice is the agent's own record of what it executed, captured by EvalRunner::captureToolCalls() from getToolResults(). When there is no record it falls back to scanning the response text for tool markers, and it says so in its reason.

The record is missing whenever the agent does not expose getToolResults(). That accessor is declared on the configuration-agent interface, so:

Agents defined in code rather than as configuration have no record.
Chat-mode targets have no tools at all.

The log carries Agent @id does not expose tool results; tool expectations fall back to reading the response text at info level. A well-behaved agent that does not echo machine-readable markers into its prose will fail every should_run: true expectation while having called the tool. There is no workaround inside ai_eval: either evaluate a configuration agent, or stop asserting on tools for that target and score the answer instead.

"argument or ordering assertion needs the agent's tool-call record"¶

An expected_tools entry used args or after, and the run had no record. Prose carries tool names in a sentence but never what a call was given or when it happened, so the expectation fails loudly rather than being scored as though it had been checked. Same cause and same options as above.

The run says nothing observed the agent¶

On an agent-mode run, ai-eval:run emits a note when no verdict came from observing the agent, and the target list marks the target Text only. Two variants:

No attached grader declares observes_actions. Of the shipped graders only tool_usage_grader and droost_state do.
One is attached but observed nothing on this run: no question carried the assertions it needs, it errored throughout, or its tool verdicts fell back to prose.

This is a note, not a warning. Scoring an agent's answer is a legitimate goal. It states a boundary: an agent that reports work it never performed scores the same as one that performed it. To measure behavior, add tool_usage_grader with expected_tools on the questions, or droost_state with metadata.droost assertions. See graders.

Gate and advisory messages¶

None of these change the verdict. They tell you what the verdict rests on.

Message	Meaning
`the gate verdict rests on judge(s) not known-good`	An LLM judge in the target's grader list is `not_validated` (never validated), `untrusted` (validated and failed), or `stale` (validated as trusted, but longer ago than `judge_trust_stale_days`, default 30). Validate it: see judges.
`the judge ... shares model family ... with the evaluated model`	Self-preference bias. The scores are not an independent measurement. Use a judge from a different family.
`judge/target model-family comparison is unknown`	Typical for agent mode, where no evaluated model ID exists at config level. Judge independence is unverified, not verified.
`N questions, M needed for +/-X margin at Y confidence`	The dataset is below the Cochran sample-size floor for `cochran_confidence` and `cochran_margin_of_error`. See scoring.

The target form refuses to save the gate threshold¶

The threshold exceeds the ceiling a run can reach. Every grader is normalized onto one shared 0 to 5 scale before averaging, so an avg_score gate above 5 can never pass, and a pass_rate gate above 1.0 can never pass.

The two metrics report this differently, so grep for the right string:

Metric	Message
`avg_score`	`Unpassable gate. @threshold exceeds the avg_score ceiling of @ceiling`
`pass_rate`	`A pass rate gate cannot exceed 100%; enter a value between 0 and 1`

The form also warns, without blocking, when an avg_score threshold falls between 0 and 1 on a scale wider than 1. That usually means a pass_rate value was typed against the avg_score metric. There is no matching warning for a small pass_rate threshold: validation returns on the pass_rate path before the warning is reached, so pass_rate: 0.1 saves silently. See scoring.

Optimizer refuses to run¶

ai-eval:optimize skips or refuses targets on grounds it always prints:

dataset too small: fewer questions than optimizer_min_dataset_size, default 15.
No dataset split: train and validation are the same question set, so any reported gain is inflated. The warning names the target; declare splits on the dataset.
below_resolution: no single answer improved, in the eyes of the judge that scored it, by at least that judge's measured resolution. --strict-resolution turns this into a refusal.
Proposer falls back to the judge model: proposer_provider and proposer_model are unset. A proposer that is also the judge scores its own proposals.

See optimizer.

Files, exports, and access¶

JSON results not written. Failed to prepare JSON results directory or Failed to write JSON results export names the path. Check Results path at /admin/config/ai/ai-eval/settings and that the web server can write it.
Failed to encode eval result JSON. Encoding is strict, with UTF-8 substitution, so a real encoding failure surfaces instead of writing a corrupt file. The log line carries the JSON error message.
Envelope export silently missing. EnvelopeExportSubscriber logs Failed to export eval run envelope with the directory or path it could not use. See sharing.
Result files disappeared after uninstall. hook_uninstall() deletes every .json file under the configured results path. Database results go with the module's tables.
Access denied on an admin page. Four permissions split the surface: operate ai eval (view results and dashboards, browse targets, run evals), administer ai eval (settings, targets, applying optimizer candidates), annotate ai eval results, and validate ai eval judges. See configuration.

Upgrade surprises¶

improvement_margin changed by itself. ai_eval_update_10005() moves it from the historical default of 0.05 to 0.4, because 0.05 sits well inside observed judge noise. An explicit override to any other value is preserved; an operator who deliberately set 0.05 is bumped.
MySQL 8 rejected an update. ai_eval_update_10004() drops an invalid TEXT column default on ai_eval_optimization.failures_json. Run drush updatedb before re-testing.
A gradient judge's average moved after upgrading. Since #3594661 a non-numeric judge score errors instead of being recorded as 0. Runs that used to be dragged down by a phantom zero now show the question as degraded instead.