Skip to content

Writing a grader plugin

How to add a grader that scores a response on a dimension ai_eval does not ship. For PHP developers; assumes you know the Drupal plugin system. For the graders that already exist and what each one sees, read the grader catalog.

A grader is a plugin class in your module's Plugin/AiEvalGrader namespace, carrying the AiEvalGrader attribute and implementing GraderInterface. It receives one question and one response and returns one GradeResult. It is discovered by ai_eval.grader_manager and becomes selectable on any evaluation target as soon as caches are cleared.

The attribute

Drupal\ai_eval\Attribute\AiEvalGrader takes eight properties.

Property Type Default Meaning
id string required The plugin ID. Targets store this string.
label string required Human-readable name, shown on the target form.
description string '' What this grader measures.
llm bool FALSE The grader scores by asking a model. Drives the judges listing and the judge-validation surface.
min_score float 0.0 The lowest score the grader can produce.
max_score float 5.0 The highest score the grader can produce.
sees_tool_calls bool FALSE An LLM judge is shown the agent's tool-call record inside its criteria block.
observes_actions bool FALSE The verdict rests on what the agent did, not on the text it wrote.

The score scale

min_score and max_score are a declaration, and Drupal\ai_eval\ScoreScale maps every score onto the shared composite 0 to 5 scale before scores are averaged. Declare the scale your grader genuinely answers on:

  • A gradient grader declares 0.0 to 5.0. Mapping is identity, so the score is passed through untouched.
  • A binary grader declares 0.0 to 1.0. A 0 maps to composite 0.0 and a 1 maps to composite 5.0. groundedness_grader is the shipped example.

Averaging a raw binary 1 with a raw gradient 4 would be meaningless, which is what the declaration exists to prevent. The declared bounds are stored alongside each score, so a result read months later is self-describing even if you later change the attribute.

sees_tool_calls

Only meaningful together with llm: TRUE. When set, LlmJudgeBase folds a rendering of the agent's tool-call record into the criteria slot of the judge prompt: tool names and arguments, never what a tool returned. Declare it when the dimension turns on what the agent did. accuracy_grader and groundedness_grader declare it; relevance_grader and actionability_grader deliberately do not, because material a judge does not need is not neutral and moves scores.

An empty record and an absent record are handled differently and must stay that way. Empty means the agent ran and called nothing, which is exactly the fact that catches an answer claiming work it never did. Absent means nobody knows, and produces no block at all.

observes_actions

This is a claim about evidence, and the runner relies on it. Two shipped graders declare it: tool_usage_grader, which reads the agent's tool-call record, and droost_state in the ai_eval_droost submodule, which ignores the response entirely and queries real Drupal state.

GraderPluginManager::actionObservingGraders() reads the flag off the plugin definitions attached to a target. The Drush run command and the targets list both use it to say plainly when an agent-mode target is scored by nothing that can see behavior. So:

What declaring observes_actions commits you to

A grader that declares it must genuinely read the tool-call record (or query real state), and must degrade honestly when no record exists. A grader that silently falls back to reading prose while still declaring the flag buys the target silence it has not earned: an agent that reports work it never performed then scores the same as one that performed it, and nothing warns the operator.

Degrading honestly means one of two things. Fail the assertion loudly and say why, which is what ToolUsageGrader does for args and after expectations that only a record can answer. Or score from the fallback and name the fallback in the reason, which is what it does for plain presence: it emits the constant ToolUsageGrader::FALLBACK_MARKER, the literal string no tool-call record available, and the run command matches on that phrase to raise the advisory anyway.

An LLM judge shown the tool record is not the same claim. It still forms its verdict out of prose, so it declares sees_tool_calls and not observes_actions.

What a grader receives

grade() takes three arguments.

Argument Type Contents
$input string The question text sent to the target. For a multi-turn question this is the flattened transcript.
$response string The target's raw response text.
$context array The question row, plus reserved keys the runner adds.

$context starts as the question row from the dataset, so expected, expected_facts, expected_tools, criteria, metadata, bundle, tags and anything else the row carries are readable under their own keys. The runner then sets these reserved keys, which always win over same-named question keys:

Key Always present Contents
response_char_limit yes Resolved char cap, from the target override, then the global default, then 4000.
messages yes The structured input as a plain role/content array, even for a single-turn question.
criteria when the question or its rubric carries any Judge criteria, after rubric llm_judge checks have been merged in.
tool_calls only when a record was read Positional list of the calls the agent executed, each with plugin_id, function_name and truncated arguments.

tool_calls is absent, not NULL, when there is no record. Test for it with array_key_exists(), never isset() or ??, so an empty record (the agent ran and called nothing) stays distinguishable from no record at all.

What a grader must return

A Drupal\ai_eval\GradeResult. Build it through GraderBase::result(), which stamps the plugin ID and the declared scale for you and clamps the score:

// A real score.
return $this->result(4.0, 'three of four facts matched');

// Not applicable to this question. Excluded from the average, silently.
return $this->result(NULL, 'no expected_tools in context, skipping');

// The grader could not run. Flags the question degraded.
return $this->result(NULL, 'provider returned no usable answer', TRUE);

The third argument is the difference between a skip and an error, and it matters. A skip is excluded from the question average without comment. An error marks the question degraded and is counted in the run's error_count. Pass TRUE only when the grader could not do its job, never when it simply does not apply.

A complete example

A deterministic grader that fails a run in which the agent called any mutating tool. It declares a binary scale because "did it write anything" has no middle, and it declares observes_actions because the verdict comes from the record. When the record is absent it errors rather than passing, since a read-only claim it cannot check is worse than a missing score.

my_eval_extras/src/Plugin/AiEvalGrader/ReadOnlyRunGrader.php:

<?php

declare(strict_types=1);

namespace Drupal\my_eval_extras\Plugin\AiEvalGrader;

use Drupal\ai_eval\Attribute\AiEvalGrader;
use Drupal\ai_eval\GradeResult;
use Drupal\ai_eval\GraderBase;

/**
 * Fails a run in which the agent called a mutating tool.
 *
 * The forbidden tool names come from the question's
 * metadata.forbidden_tool_prefixes, defaulting to the usual write verbs.
 */
#[AiEvalGrader(
  id: 'read_only_run',
  label: 'Read-only run',
  description: 'Deterministic: did the agent call any mutating tool while answering?',
  llm: FALSE,
  min_score: 0.0,
  max_score: 1.0,
  observes_actions: TRUE,
)]
final class ReadOnlyRunGrader extends GraderBase {

  /**
   * Name fragments that mark a tool as mutating.
   */
  private const DEFAULT_PREFIXES = ['create_', 'update_', 'delete_', 'edit_', 'save_'];

  /**
   * {@inheritdoc}
   */
  public function grade(string $input, string $response, array $context): GradeResult {
    // array_key_exists, not isset: an empty record is a real answer (the agent
    // ran and called nothing) and must not be confused with no record at all.
    if (!array_key_exists('tool_calls', $context) || !is_array($context['tool_calls'])) {
      // This grader declares observes_actions. On a run with no record it
      // cannot do what it declared, so it errors instead of quietly passing.
      return $this->result(NULL, 'no tool-call record available, cannot verify the run was read-only', TRUE);
    }

    $prefixes = $this->prefixes($context);
    $offenders = [];
    foreach ($context['tool_calls'] as $call) {
      if (!is_array($call)) {
        continue;
      }
      $name = is_string($call['function_name'] ?? NULL) && $call['function_name'] !== ''
        ? $call['function_name']
        : (string) ($call['plugin_id'] ?? '');
      if ($name === '') {
        continue;
      }
      foreach ($prefixes as $prefix) {
        if (str_contains($name, $prefix)) {
          $offenders[$name] = TRUE;
          break;
        }
      }
    }

    if ($offenders !== []) {
      return $this->result(0.0, sprintf(
        'mutating tools called: %s (from the agent tool-call record)',
        implode(', ', array_keys($offenders)),
      ));
    }

    return $this->result(1.0, sprintf(
      'no mutating tool among %d recorded calls (from the agent tool-call record)',
      count($context['tool_calls']),
    ));
  }

  /**
   * The name fragments that mark a tool as mutating, for this question.
   *
   * @param array<string, mixed> $context
   *   The grader context.
   *
   * @return array<int, string>
   *   Non-empty list of lowercase fragments.
   */
  private function prefixes(array $context): array {
    $configured = $context['metadata']['forbidden_tool_prefixes'] ?? NULL;
    if (!is_array($configured)) {
      return self::DEFAULT_PREFIXES;
    }
    $fragments = array_values(array_filter(
      array_map(static fn(mixed $v): string => is_string($v) ? strtolower(trim($v)) : '', $configured),
      static fn(string $v): bool => $v !== '',
    ));

    return $fragments === [] ? self::DEFAULT_PREFIXES : $fragments;
  }

}

Clear caches and the grader appears on the target form. The checkbox label is the grader label followed by its scale, so this one reads "Read-only run (binary, 0-1)".

Injecting services

Implement ContainerFactoryPluginInterface and add a create() factory. The signature is fixed by the interface; keep the three plugin arguments first and call the parent constructor:

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

final class MyGrader extends GraderBase implements ContainerFactoryPluginInterface {

  public function __construct(
    array $configuration,
    string $plugin_id,
    mixed $plugin_definition,
    private readonly LoggerInterface $channelLogger,
  ) {
    parent::__construct($configuration, $plugin_id, $plugin_definition);
  }

  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition): static {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $container->get('logger.channel.ai_eval'),
    );
  }

}

Writing an LLM judge instead

Extend Drupal\ai_eval\LlmJudgeBase rather than GraderBase. It handles the provider call, the rate limiter, deterministic decoding (temperature pinned to 0), stored prompt overrides, response truncation and JSON parsing. It already implements ContainerFactoryPluginInterface, so do not write your own create() unless you also call the parent one.

Three abstract methods:

<?php

declare(strict_types=1);

namespace Drupal\my_eval_extras\Plugin\AiEvalGrader;

use Drupal\ai_eval\Attribute\AiEvalGrader;
use Drupal\ai_eval\LlmJudgeBase;

/**
 * LLM judge scoring how well a response matches the site's tone of voice.
 */
#[AiEvalGrader(
  id: 'house_tone',
  label: 'House tone',
  description: 'LLM judge: does the response match the documented house tone (1-5)?',
  llm: TRUE,
  min_score: 0.0,
  max_score: 5.0,
)]
final class HouseToneGrader extends LlmJudgeBase {

  /**
   * {@inheritdoc}
   */
  protected function getDimension(): string {
    return 'house_tone';
  }

  /**
   * {@inheritdoc}
   */
  protected function getDimensionDescription(): string {
    return 'adherence to the house tone of voice: plain, direct, second person, no marketing language';
  }

  /**
   * {@inheritdoc}
   */
  protected function getScoringGuidance(): string {
    return <<<GUIDANCE
    5: Plain and direct throughout, second person, no marketing language.
    4: One lapse into promotional or vague phrasing.
    3: Several lapses, or drifts into third person.
    2: Mostly promotional or hedged.
    1: Reads as marketing copy.
    GUIDANCE;
  }

}

getDimension() is what gold labels and judge validation key on, so pick a stable token and do not change it once labels exist.

Two hooks worth knowing when you need something other than a 1 to 5 gradient:

  • defaultTemplate() returns the prompt template. Override it to ask a different question, as GroundednessGrader does for its binary prompt. The template is filled by positional sprintf with four values in this order: dimension description, question, criteria followed by your getScoringGuidance() text, response. The third slot carries both, joined by a newline, which is where your scoring guidance reaches the model. Do not add a fifth placeholder; stored prompt overrides are positional too and would break.
  • interpretJudgeScore(float $raw): ?float turns the number the judge returned into a score on your scale. The default clamps to the declared bounds. Return NULL to mark the answer unusable, which becomes an errored result rather than a score of zero. GroundednessGrader returns NULL for anything outside 0 to 1, because a judge that graded on a scale it was not asked about did not answer the question.

If your judge needs extra material in the prompt, fold it into $context['criteria'] and call parent::grade(), the way FactMatchGrader prepends its ground-truth block. That keeps stored overrides working.