Events¶

The events ai_eval dispatches, and how to subscribe to one. For PHP developers who want to react to a finished eval run: post results to a dashboard, open an issue on a regression, notify a channel.

ai_eval dispatches one event. It invokes two hooks, both plugin-definition alters, documented in ai_eval.api.php at the module root and summarized in the extension points overview.

EvalRunCompleteEvent¶

Drupal\ai_eval\Event\EvalRunCompleteEvent, a Drupal\Component\EventDispatcher\Event.

When it fires¶

Once per completed run, immediately after the ai_eval_result row has been inserted. Two call sites dispatch it, and between them they cover every way a run can finish. Only the EvalRunner path also marks an ai_eval_run row complete; the Drush path writes no run row at all, which is why it carries no run_id.

Service\EvalRunner::finalizeRun(), which is the path the admin UI and batched runs take.
The ai-eval:run Drush command, once per target it evaluated.

Exactly once per run. An incremental run that starts, steps many times and then finalizes dispatches one event, and a second finalizeRun() call on an already-finalized run dispatches nothing.

It does not fire for a run that failed before producing a result row, or for a cancelled run.

flowchart LR
  R["Run finishes"] --> I["INSERT ai_eval_result"]
  I --> U["Run row marked complete (EvalRunner path only)"]
  U --> D["dispatch(EvalRunCompleteEvent)"]
  D --> S1["EnvelopeExportSubscriber"]
  D --> S2["Your subscriber"]

What it carries¶

One readonly property, $resultRow, a \stdClass shaped like the inserted database row: the inserted id plus every column value that was written.

Property	Type	Notes
`id`	int/string	The inserted `ai_eval_result` row ID.
`target_id`	string	The evaluation target's ID.
`run_id`	int	Present on the `EvalRunner` path. The Drush command writes no `run_id`.
`timestamp`	int	Unix time the row was written.
`avg_score`	float	Composite average on the 0 to 5 scale, over the successfully scored questions only.
`pass_rate`	float	Fraction of the successfully scored questions that passed. The denominator is `question_count - error_count`, not `question_count`.
`pass_rate_ci_low`	float/null	Lower bound of the pass-rate confidence interval.
`pass_rate_ci_high`	float/null	Upper bound.
`passed_gate`	int	1 or 0.
`question_count`	int	Questions attempted, errored ones included. Both dispatch sites store `count($questionScores)`, so this is not a count of successfully scored questions. Subtract `error_count` for that.
`scores_json`	string	JSON blob of per-question scores.
`source`	string	How the run was started, for example `manual`.
`duration`	int/float	Seconds. The `EvalRunner` path computes whole seconds from timestamps; the Drush command reports elapsed wall time rounded to two decimals. Cast it, do not assume an integer.
`error_count`	int	Questions that could not be scored: a failed target invocation, invalid input, or an errored grader. Not only grader errors.
`config_json`	string/null	The run identity snapshot, when one was captured.

Read defensively

run_id is written by the EvalRunner path and not by the Drush command, so a subscriber that must work on both should check for it rather than assume it. Everything else in the table is written by both.

A working subscriber¶

Log a warning when a run's composite average drops more than half a point below the previous run for the same target.

my_eval_extras/src/EventSubscriber/ScoreRegressionSubscriber.php:

<?php

declare(strict_types=1);

namespace Drupal\my_eval_extras\EventSubscriber;

use Drupal\ai_eval\Event\EvalRunCompleteEvent;
use Drupal\Core\Database\Connection;
use Psr\Log\LoggerInterface;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;

/**
 * Warns when a target's composite average drops against its previous run.
 */
final class ScoreRegressionSubscriber implements EventSubscriberInterface {

  /**
   * How far the average may fall before it is worth a warning.
   */
  private const TOLERANCE = 0.5;

  public function __construct(
    private readonly Connection $database,
    private readonly LoggerInterface $logger,
  ) {}

  /**
   * {@inheritdoc}
   */
  public static function getSubscribedEvents(): array {
    return [
      EvalRunCompleteEvent::class => 'onRunComplete',
    ];
  }

  /**
   * Compares the completed run against the previous one for the same target.
   *
   * @param \Drupal\ai_eval\Event\EvalRunCompleteEvent $event
   *   The run-complete event carrying the inserted result row.
   */
  public function onRunComplete(EvalRunCompleteEvent $event): void {
    // Never let a subscriber break the run that produced the result.
    try {
      $row = $event->resultRow;

      $previous = $this->database->select('ai_eval_result', 'r')
        ->fields('r', ['avg_score'])
        ->condition('target_id', (string) $row->target_id)
        ->condition('id', (int) $row->id, '<')
        ->orderBy('id', 'DESC')
        ->range(0, 1)
        ->execute()
        ->fetchField();

      if ($previous === FALSE || $previous === NULL) {
        return;
      }

      $drop = (float) $previous - (float) $row->avg_score;
      if ($drop <= self::TOLERANCE) {
        return;
      }

      $this->logger->warning('Target @target dropped @drop points, from @before to @after over @count questions.', [
        '@target' => (string) $row->target_id,
        '@drop' => round($drop, 2),
        '@before' => round((float) $previous, 2),
        '@after' => round((float) $row->avg_score, 2),
        '@count' => (int) $row->question_count,
      ]);
    }
    catch (\Throwable $e) {
      $this->logger->error('Score regression check failed: @message', [
        '@message' => $e->getMessage(),
      ]);
    }
  }

}

my_eval_extras.services.yml, at your module root:

services:
  my_eval_extras.score_regression_subscriber:
    class: Drupal\my_eval_extras\EventSubscriber\ScoreRegressionSubscriber
    arguments:
      - '@database'
      - '@logger.channel.ai_eval'
    tags:
      - { name: event_subscriber }

Do not break the run¶

The dispatch happens inside the run's own execution path, so an exception thrown from a subscriber propagates into the code that just finished the run. Catch \Throwable and log it, as the example does. The shipped EnvelopeExportSubscriber follows the same rule: its errors are logged, never thrown, so a failed export never disrupts the eval run that produced it.

If the work you want to do is slow, queue it from the subscriber rather than doing it there.

The shipped subscriber¶

Drupal\ai_eval\EventSubscriber\EnvelopeExportSubscriber writes a portable result envelope when a run completes, gated behind the ai_eval.settings:export_envelope_on_complete flag, which is FALSE by default. Read it as a reference implementation, and see sharing results for what an envelope is.

ai_eval.api.php at the module root, for the two alter hooks.
Extension points overview.