Skip to content

Writing a check executor

How to make a new rubric check kind runnable, and how to teach the rubric validator about it so authors get errors at validation time instead of silence at run time. For PHP developers. For rubric authoring, read rubrics.

A check executor is an ordinary service tagged ai_eval.check_executor. It is not a plugin-manager plugin: there is no attribute and no Plugin/ namespace requirement, though the shipped ones live in Plugin/CheckExecutor by convention.

How a check reaches an executor

flowchart TD
  Q["Question with rubric_ref"] --> RCG["rubric_checks grader"]
  RCG --> RCE["RubricCheckEvaluator"]
  RCE --> L["RubricLoader resolves the ref"]
  L --> C["Each check in order"]
  C --> F{"First executor whose<br/>applies() is TRUE"}
  F -->|found| X["execute() returns a CheckOutcome"]
  F -->|none| S["Non-executable:<br/>excluded from combining,<br/>named in the reason"]
  X --> CB["Combine per scoring.combine"]
  CB --> G["One GradeResult"]

RubricCheckEvaluator walks the rubric's checks in order and hands each to the first tagged executor whose applies() returns TRUE. Order of the tagged collection therefore matters if two executors claim the same kind; pin the kind with an exact comparison and the question does not arise.

When zero checks in a rubric were executable, the evaluator returns NULL and the rubric_checks grader records a skip rather than a zero.

Which kinds are runner-executable

The rubric schema is a portable contract, so it describes more kinds than ai_eval executes. RubricCheckEvaluator names the non-executable ones in its docblock, and this is the current split.

Kind Executed by
must_contain_any StringCheckExecutor
must_not_contain StringCheckExecutor
regex StringCheckExecutor
target_match TargetMatchExecutor
composite RubricCheckEvaluator itself, recursively

Declarative kinds with no executor, by design or pending one:

Kind Why there is no executor
llm_judge Never executed here, and the judge prompt it names is never resolved. EvalRunner::mergeRubricCriteria() appends only the check's optional free-text description to the criteria the LLM judge graders already read. judge_prompt_ref, judges, aggregation and the check's own threshold are not read by any production code. A check with no description contributes nothing at all, which is true of both shipped llm_judge fixtures. Write a description if you want the judge to see anything.
tool_usage No executor, and none planned. The implemented tool assertion is case-level expected_tools, scored by ToolUsageGrader against the agent's own record. The rubric kind stays in the schema as contract vocabulary for runners that express a tool assertion at rubric level rather than case level.
command Executor pending: it needs the sandbox harness, and must run inside the declared sandbox rather than on the host.
score_delta Executor pending behind the ai_bench measurement-dict shape.

Other kinds in schema/rubric.schema.json, such as json_schema, php_lint, markdown_structure, format and fact_match, are schema-valid and have no executor in ai_eval today. A rubric using one validates and its check is skipped, counted in the combined reason as not executable.

composite is handled by the evaluator, not by an executor. It resolves the named sub-rubric, evaluates it recursively, and turns the sub-verdict into a single outcome of kind composite. Recursion is cycle-guarded and capped at depth 3.

The interface

Drupal\ai_eval\CheckExecutorInterface has two methods.

Method Returns Contract
applies(array $check) bool Whether execute() can handle this check. The check always carries a kind key.
execute(array $check, string $response, array $context) CheckOutcome Run the check. $context is the question row.

CheckOutcome is a value object with five constructor arguments: the check id (may be an empty string), the kind, a score, a pass flag and a reason. The score is clamped to 0 to 1, which is the rubric scale. The 0 to 5 composite scale is the grader's concern, not the check's.

Two conventions the shipped executors follow and yours should too.

Report a continuous score where you have one. A binary check scores 1.0 or 0.0. A similarity check reports the real number, so weighted_avg, min, max and median combining sees 0.85 rather than 0.0, while pass still gates on the check's own threshold for all_pass and any_pass.

Fail authoring errors loudly. An unresolvable reference, an invalid regex, an unknown mode, a malformed value: fail the check with the cause in the reason. Never skip silently. StringCheckExecutor sets the precedent by failing on a pattern that does not compile, and TargetMatchExecutor follows it for every one of its authoring errors. An author needs to see the mistake in the results.

Contributing the schema for your kind

Add CheckSchemaProviderInterface to the same executor and RubricValidator will accept your kind. It returns one JSON Schema definition per kind, keyed by kind token; each definition is spliced in at $defs.check_<kind> and referenced from the check oneOf.

The rules, which the validator enforces or relies on:

  • Pin the kind with a const and list kind as required, so your definition can never swallow another kind's checks.
  • A kind whose definition already exists is ignored. Core kinds cannot be overridden, and between two modules the first to register wins.
  • Follow the core shape: allOf with a $ref to #/$defs/check_base plus your own properties, and unevaluatedProperties: false on the outer definition, so typos in a check are rejected.
  • A rubric using your kind validates only where your module is installed. On a site without it, validation rejects the kind exactly as it rejects a typo.

The same tagged service both provides the schema and runs the check, so the schema Drupal validates against and the runtime dispatch cannot drift apart.

Editors are a separate matter

The splice happens in PHP inside a running site. Nothing exports the combined schema, and the yaml-language-server URL below serves the static schema/rubric.schema.json, which cannot know about your kind. So an editor pointed at that URL flags a module-contributed kind as invalid while Drupal validates and runs it happily. Drop the schema comment from rubrics that use your own kinds, or expect the editor to complain.

A complete example

An executor for a word_count kind that asserts the response length in words falls inside a range, and contributes its own schema definition.

my_eval_extras/src/CheckExecutor/WordCountExecutor.php:

<?php

declare(strict_types=1);

namespace Drupal\my_eval_extras\CheckExecutor;

use Drupal\ai_eval\CheckExecutorInterface;
use Drupal\ai_eval\CheckOutcome;
use Drupal\ai_eval\CheckSchemaProviderInterface;

/**
 * Executes the word_count check kind: response length within a word range.
 *
 * Pure and dependency-free. Scores 1.0 inside the range and 0.0 outside it;
 * a range is a hard boundary, so there is no meaningful partial credit.
 * Authoring errors fail the check with the cause in the reason.
 */
final class WordCountExecutor implements CheckExecutorInterface, CheckSchemaProviderInterface {

  /**
   * {@inheritdoc}
   */
  public function applies(array $check): bool {
    return ($check['kind'] ?? '') === 'word_count';
  }

  /**
   * {@inheritdoc}
   */
  public function execute(array $check, string $response, array $context): CheckOutcome {
    $id = (string) ($check['id'] ?? '');

    $min = $check['min'] ?? 0;
    $max = $check['max'] ?? NULL;
    if (!is_int($min) || $min < 0 || ($max !== NULL && (!is_int($max) || $max < $min))) {
      return new CheckOutcome($id, 'word_count', 0.0, FALSE, 'invalid min/max on the check');
    }

    $words = preg_split('/\s+/u', trim($response), -1, PREG_SPLIT_NO_EMPTY);
    if ($words === FALSE) {
      return new CheckOutcome($id, 'word_count', 0.0, FALSE, 'response could not be split into words');
    }
    $count = count($words);

    if ($count < $min) {
      return new CheckOutcome($id, 'word_count', 0.0, FALSE, sprintf('%d words, minimum %d', $count, $min));
    }
    if ($max !== NULL && $count > $max) {
      return new CheckOutcome($id, 'word_count', 0.0, FALSE, sprintf('%d words, maximum %d', $count, $max));
    }

    return new CheckOutcome($id, 'word_count', 1.0, TRUE, sprintf('%d words, within range', $count));
  }

  /**
   * {@inheritdoc}
   */
  public function checkSchemaDefinitions(): array {
    return [
      'word_count' => [
        'allOf' => [
          ['$ref' => '#/$defs/check_base'],
          [
            'type' => 'object',
            'required' => ['kind'],
            'properties' => [
              'kind' => ['const' => 'word_count'],
              'min' => ['type' => 'integer', 'minimum' => 0],
              'max' => ['type' => 'integer', 'minimum' => 0],
            ],
          ],
        ],
        'unevaluatedProperties' => FALSE,
      ],
    ];
  }

}

Register it in my_eval_extras.services.yml:

services:
  my_eval_extras.check_executor.word_count:
    class: Drupal\my_eval_extras\CheckExecutor\WordCountExecutor
    tags:
      - { name: ai_eval.check_executor }

Clear caches, and Drupal both validates and runs this rubric. Note the absence of a yaml-language-server line: the published schema does not carry your kind, so an editor would flag it even though the site accepts it.

id: summary_shape
version: "1.0.0"
checks:
  - id: not_a_wall_of_text
    kind: word_count
    min: 20
    max: 150
  - id: cites_the_node
    kind: must_contain_any
    values: ["/node/"]
scoring:
  combine: all_pass

Injecting services

Tagged services are ordinary services, so constructor injection is plain arguments: in your services.yml. TargetMatchExecutor takes two:

  my_eval_extras.check_executor.similarity:
    class: Drupal\my_eval_extras\CheckExecutor\SimilarityExecutor
    arguments:
      - '@ai_eval.chrf_scorer'
      - '@ai_eval.check_template_resolver'
    tags:
      - { name: ai_eval.check_executor }

ai_eval.check_template_resolver resolves a bare dot-path such as expected.reference or metadata.target_title against the question, and can return one string or a list of acceptable variants. Use it rather than reading $context by hand, so your kind's reference_path behaves like the shipped one.

Testing

tests/modules/ai_eval_check_schema_test is a working fixture module: a single executor that registers a probe_fixture kind and deliberately also offers a colliding definition for a core kind, so the tests can prove core stays canonical. Copy its shape.