FAQ

Short answers, each pointing at the page that covers the topic properly. For a symptom you are looking at right now, go to troubleshooting instead.

Does this replace manual QA?

No. It measures what its graders can see. Of the graders that ship, only tool_usage_grader and droost_state form a verdict from something other than the response text, and an LLM judge's verdict is only worth what its validation against human labels says it is. See graders and judges.

Does it work without the AI Agents module?

Yes, in chat mode. ai_agents is a Composer suggestion, not a dependency, and the runner only needs it for agent-mode targets, where it throws if the module is absent. See targets.

Can I use a local model?

Yes. The module never talks to a model directly; it resolves both the target provider and the judge through the AI module's provider manager, so any provider plugin installed on the site works, including one that points at a locally hosted model. See configuration.

What LLM providers work as judges?

Any provider plugin the AI module has installed. The judge is stored as judge_provider plus judge_model, and the settings form writes them from a single provider__model value. See configuration.

How many questions does a dataset need?

Enough that the pass rate means something. ai-eval:run computes the Cochran floor from cochran_confidence (default 0.95) and cochran_margin_of_error (default 0.05) and warns when the dataset is below it, and the optimizer refuses targets under optimizer_min_dataset_size (default 15). See scoring.

How much does a run cost?

The module does not estimate or cap spend. Each question costs one target call plus one provider call per LLM judge that applies to it (a judge that skips costs nothing), and an agent-mode target costs however many calls its agent loop makes. Provider-reported input and output token counts are recorded per question when the provider reports them, and they travel in the exported envelope. See sharing.

How long does a run take?

The floor is set by throttling, not by the models. rate_limit_delay_between_calls (default 2.0 seconds) is slept before every provider call, including every judge call, so ten questions with four LLM judges spend at least 100 seconds asleep before any model has answered. See running.

Are judge scores reproducible?

Partly. The judge pins temperature to 0 and top_p to 1 whenever the provider advertises those knobs, and records which knobs it actually set, so a run says whether the pin took effect. A provider advertising neither is scored at its own defaults, and no pin makes a hosted model deterministic. See judges.

Can I run it in CI?

Yes. drush ai-eval:run exits 1 when any target with gate: hard fails its gate, and 0 when all hard gates pass, so it works as a build step unchanged. --json also writes the full results to the configured results path. See running.

What happens to data sent to a third-party provider?

An LLM judge's prompt carries the question input, the criteria, and the model's response truncated to judge_response_char_limit (default 4000 characters). Judges that declare sees_tool_calls also receive the run's tool-call record, including argument values, which are reduced to short scalars and redacted when the argument name looks like a credential. Name-based redaction reduces exposure rather than removing it. See configuration.

Can I add my own graders?

Yes, from any module. Graders are plugins discovered by the AiEvalGrader attribute, and dataset sources, check executors and judges are extensible the same way. See writing a grader.

What happens to results when I uninstall the module?

hook_uninstall() deletes every .json file under the configured results path. Results stored in the database go when the module's tables are dropped. Nothing is archived first. See sharing.