Running evaluations

Three ways to run a target: the admin UI, the ai-eval:run Drush command, and a scheduled CI job. All three execute the same run engine and write the same result row. This page is for whoever operates the evals.

From the UI

The Targets page at /admin/config/ai/ai-eval has a Run button on every enabled target. Pressing it starts a run and swaps the button for a progress bar.

The run advances one question per HTTP request, so a hundred-question dataset is a hundred small requests rather than one that times out. Progress and the error count update on each poll, and a Cancel button stops the run cleanly. A cancelled run is marked cancelled and writes no result row, so its partial scores are not recorded.

Two locks keep concurrent operators honest: a per-target lock at run start, and a per-run lock on each step. The same target cannot be started twice at once, and no question is graded twice.

stateDiagram-v2
    [*] --> pending: Run pressed
    pending --> running: first step
    running --> running: one question per poll
    running --> complete: last question graded
    running --> cancelled: Cancel pressed
    running --> error: run failed
    complete --> [*]: result row written
    cancelled --> [*]: no result row

Running from the UI needs the operate ai eval or administer ai eval permission.

From Drush

The command is ai-eval:run, alias aer. With no options it runs every enabled target.

# Every enabled target
drush ai-eval:run

# One target, or one question of one target
drush ai-eval:run --target=my_target
drush ai-eval:run --target=my_target --question=Q01

# List what would run, calling no provider
drush ai-eval:run --dry-run

# Also write a timestamped JSON results file
drush ai-eval:run --json

# Tag the run so cron runs are distinguishable from manual ones
drush ai-eval:run --source=cron

# Run every question as Drupal user 2
drush ai-eval:run --as-user=2

Options

Option	Default	What it does
--target	all enabled targets	Run only this target id. An unknown id is an error that lists the available ids.
--question	all questions	Run only this question id. Skips the sample-size advisory, since the run is a single question.
--split	all	Run only the train, validation, or test partition of the dataset. Any other value is an error. See datasets.
--source	manual	The run source tag stored on the result row.
--as-user	none	The Drupal uid used as caller for every question.
--dry-run	off	List targets and question counts, then exit. Ignores --split, see below.
--json	off	Also write a JSON results file.

Dry run

--dry-run resolves each target's dataset and prints the target id, the question count, and the dataset reference. It calls no AI provider, grades nothing, and writes no result row, so it costs nothing and is the cheap way to confirm that a target points at the dataset you think it does.

$ drush ai-eval:run --dry-run
DRY RUN: would evaluate these targets:
  my_target: 24 questions from my_dataset.yml

Warning

--dry-run ignores --split. The dry-run branch loads and counts the dataset before the split reaches the runner configuration, so --dry-run --split=test reports the whole dataset, not the test partition. The count it prints is the full question count in that case.

The JSON results file

--json writes one file per invocation, covering every target the run touched. It goes to the results_path set in configuration, or to the system temp directory plus /ai_eval_results when that setting is empty. The filename is results-YYYYmmdd-HHMMSS.json and the payload is the run timestamp plus one entry per target id. The path is printed at the end of the run.

This file is a raw dump of the run, not a portable artifact. For an artifact meant to travel between sites, see sharing results.

Tagging the run source

--source sets the source column on the result row, and nothing else. It defaults to manual, the UI writes ui, and a scheduled job should pass --source=cron. The tag is carried into exported envelopes as run_source, so "did this number come from a nightly job or from someone pressing Run" stays answerable after the fact.

Running as a specific user

--as-user=N sets the run's default caller uid. A question that carries its own caller_uid keeps it, so the option is a default rather than an override.

Warning

The option only has an effect if a host module registers a ai_eval.caller_context service implementing Drupal\ai_eval\Service\CallerContextInterface. AI Eval declares that dependency as optional and ships no implementation of its own. Without a host implementation, the uid is resolved and then handed to nothing: --as-user changes nothing about the run. It exists for sites whose agents resolve user-scoped state (per-user memory, per-user access) and that wire up the service themselves.

Advisories printed during a run

A run prints warnings that never change the verdict or the exit code:

• The dataset is below the Cochran sample-size floor. See scoring.
• The gate verdict rests on a judge that is untrusted, stale, or never validated. See judges.
• The judge shares a model family with the evaluated model, so the scores are not an independent measurement.
• The target runs in agent mode and no attached grader observed what the agent did, so every score is a reading of the response text. See graders.

Exit codes

Code	Meaning
0	Every hard gate passed, or the run was a dry run.
1	A hard gate failed, or the command could not run: unknown target, no targets configured, an invalid --split, or a failed JSON write.

A soft gate that misses its threshold logs a warning and leaves the exit code at 0. Which targets are hard and which are soft is a per-target setting, covered in scoring.

In CI

Because a failed hard gate exits 1, a scheduled pipeline job can gate on eval health. AI Eval ships no cron integration on purpose: a full run can be hundreds of LLM calls, and when to spend that belongs to site operations.

# .gitlab-ci.yml, run from a scheduled pipeline
ai_eval_nightly:
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  script:
    - drush ai-eval:run --source=cron --target=my_target --json
  artifacts:
    when: always
    paths:
      - sites/default/files/ai-eval-results/
    expire_in: 30 days

Point the paths entry at whatever results_path you configured, and create the schedule under CI/CD, Schedules in the project settings. A crontab line does the same job on a host with no pipeline:

0 2 * * * cd /var/www/html && drush ai-eval:run --source=cron

Two things are worth adding to a pipeline that gates on eval results. Run drush ai-eval:judges --json in the same job to record which judges were trusted at the time, since an eval result is only as good as the judges behind it; see judges. And export an envelope if you want the run's evidence to outlive the pipeline log; see sharing results.