Skip to content

Writing a dataset source plugin

How to load eval questions from a store ai_eval does not ship a reader for: a remote registry, a git checkout, a third-party test-case API. For PHP developers. For the dataset file format itself, read datasets.

A dataset source is a plugin class in your module's Plugin/AiEvalDatasetSource namespace, carrying the AiEvalDatasetSource attribute and implementing DatasetSourceInterface, normally by extending DatasetSourceBase. ai_eval ships three: file, config and entity.

How a source is reached

An evaluation target names a source plugin ID and a source-relative reference. Service\DatasetLoader is a thin resolver that instantiates the named plugin and calls it. Everything about parsing, validating and listing belongs to the plugin.

flowchart LR
  T["Evaluation target<br/>source id plus ref"] --> DL["DatasetLoader"]
  DL --> PM["DatasetSourcePluginManager"]
  PM --> P["Your source plugin"]
  P --> ST[("Your store")]
  P --> Q["Question rows"]
  Q --> R["EvalRunner"]

An unknown source ID is logged and returns an empty question list. It never throws.

The attribute

Drupal\ai_eval\Attribute\AiEvalDatasetSource takes three properties.

Property Type Default Meaning
id string required The plugin ID. Targets store this string.
label string required Human-readable name, shown in source selectors.
description string '' What this source loads from.

The interface

Drupal\ai_eval\DatasetSourceInterface has four methods. DatasetSourceBase implements the last two, so a read-only source normally writes two.

Method Returns Contract
load(string $ref, ?string $questionId = NULL) array<int, array<string, mixed>> Question rows for one dataset reference, re-indexed. Disabled questions excluded. Filter to one question when $questionId is given.
listDatasets() array<int, string> The source-relative references this source can load.
listDatasetsWithLabels() array<string, string> Map of reference to human label. Base returns ref => ref.
capabilities() array<int, string> Subset of read, write, promote. Base returns ['read']. Always contains read.

load() must not throw. A reference that does not resolve, a parse failure, or a document that fails schema validation returns an empty array, and should log a warning on the ai_eval log channel so the empty result is explainable.

The shipped file source returns silently on every one of those cases, which is behavior to improve on rather than copy: a run against a mistyped path reports zero questions with nothing in the log saying why.

DatasetSourceBase::normalizeQuestions() does the common post-processing for you: it drops rows that are not arrays (logging each one), drops rows with a truthy disabled, applies the $questionId filter, and re-indexes. Call it with the raw rows, a label for log messages, the question ID and a logger.

Optional capability interfaces

These are separate interfaces, not methods on DatasetSourceInterface, so an existing third-party source stays valid without changes.

Interface Adds Implemented by
WritableDatasetSourceInterface appendQuestions(), createDataset() entity
SplitAwareDatasetSourceInterface loadSplitsConfig() file, config

WritableDatasetSourceInterface is the single seam that promotion, synthetic generation and any future writer funnel through. appendQuestions() validates every question against the dataset JSON Schema before persisting anything, and persists none of them if any one fails. It allocates non-colliding question IDs and returns them in input order. Detect support with instanceof WritableDatasetSourceInterface for type safety, or with capabilities() when the UI just needs to know what to offer.

SplitAwareDatasetSourceInterface exposes the dataset-level splits block, which load() cannot return because it yields question rows only. Implement it when your backing store holds the whole dataset document. Sources without document-level metadata simply do not implement it, and only explicit per-question split labels then apply. Shape validation of the block is DatasetSplitter's job, not yours: return the raw block or [].

A complete example

A read-only source that fetches dataset documents over HTTP from a registry, validates each against the shipped dataset schema, and caches the parsed result for the request.

my_eval_extras/src/Plugin/AiEvalDatasetSource/RegistrySource.php:

<?php

declare(strict_types=1);

namespace Drupal\my_eval_extras\Plugin\AiEvalDatasetSource;

use Drupal\ai_eval\Attribute\AiEvalDatasetSource;
use Drupal\ai_eval\DatasetSourceBase;
use Drupal\ai_eval\Service\DatasetValidator;
use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use GuzzleHttp\ClientInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\Yaml\Yaml;

/**
 * Loads eval datasets from a remote Eval Commons registry over HTTP.
 *
 * The ref is a bare document name; the base URL comes from
 * my_eval_extras.settings:registry_base_url. Documents are validated against
 * the ai_eval dataset JSON Schema before their questions are returned, so a
 * registry serving something else cannot poison a run.
 */
#[AiEvalDatasetSource(
  id: 'registry',
  label: 'Registry',
  description: 'Loads datasets from a remote Eval Commons registry over HTTP.',
)]
final class RegistrySource extends DatasetSourceBase implements ContainerFactoryPluginInterface {

  /**
   * Parsed documents, keyed by ref, for the life of the request.
   *
   * @var array<string, array<string, mixed>|null>
   */
  private array $cache = [];

  public function __construct(
    array $configuration,
    string $plugin_id,
    mixed $plugin_definition,
    private readonly ClientInterface $httpClient,
    private readonly ConfigFactoryInterface $configFactory,
    private readonly DatasetValidator $datasetValidator,
    private readonly LoggerInterface $channelLogger,
  ) {
    parent::__construct($configuration, $plugin_id, $plugin_definition);
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition): static {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $container->get('http_client'),
      $container->get('config.factory'),
      $container->get('ai_eval.dataset_validator'),
      $container->get('logger.channel.ai_eval'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function load(string $ref, ?string $questionId = NULL): array {
    $document = $this->fetch($ref);
    if ($document === NULL) {
      return [];
    }

    return $this->normalizeQuestions(
      $document['questions'] ?? [],
      'registry:' . $ref,
      $questionId,
      $this->channelLogger,
    );
  }

  /**
   * {@inheritdoc}
   */
  public function listDatasets(): array {
    $index = $this->configFactory
      ->get('my_eval_extras.settings')
      ->get('registry_documents');

    return is_array($index) ? array_values(array_filter($index, 'is_string')) : [];
  }

  /**
   * Fetches and validates one registry document.
   *
   * Never throws: a run must degrade to zero questions with a log line, not
   * abort because a registry was briefly unreachable.
   *
   * @param string $ref
   *   The document name.
   *
   * @return array<string, mixed>|null
   *   The validated document, or NULL when it could not be loaded.
   */
  private function fetch(string $ref): ?array {
    if (array_key_exists($ref, $this->cache)) {
      return $this->cache[$ref];
    }
    $this->cache[$ref] = NULL;

    // A ref is a bare document name. Anything path-like is refused rather
    // than sent, so a crafted target cannot reach outside the registry.
    if ($ref === '' || basename($ref) !== $ref) {
      $this->channelLogger->warning('RegistrySource: refusing non-simple dataset ref @ref.', ['@ref' => $ref]);
      return NULL;
    }

    $base = rtrim((string) $this->configFactory->get('my_eval_extras.settings')->get('registry_base_url'), '/');
    if ($base === '') {
      $this->channelLogger->warning('RegistrySource: no registry_base_url configured.');
      return NULL;
    }

    try {
      $body = (string) $this->httpClient
        ->request('GET', $base . '/' . $ref, ['timeout' => 10])
        ->getBody();
      $document = Yaml::parse($body);
    }
    catch (\Throwable $e) {
      $this->channelLogger->warning('RegistrySource: could not load @ref: @message', [
        '@ref' => $ref,
        '@message' => $e->getMessage(),
      ]);
      return NULL;
    }

    if (!is_array($document) || !$this->datasetValidator->validate($document)) {
      $this->channelLogger->warning('RegistrySource: @ref failed dataset schema validation: @errors', [
        '@ref' => $ref,
        '@errors' => implode('; ', $this->datasetValidator->errors()),
      ]);
      return NULL;
    }

    $this->cache[$ref] = $document;
    return $document;
  }

}

Clear caches and registry appears as a dataset source on the target form.

Validate what you load

Two of the three shipped sources, config and entity, validate the document with ai_eval.dataset_validator before returning any questions, and return an empty array with a log line when validation fails. Do the same. A source is the boundary between a store you do not fully control and a run whose numbers people will quote.

The default source does not validate

file is the shipped default for every target (EvalTarget::$dataset_source), and FileSource::load() parses the YAML and hands the rows straight on without calling the validator. A malformed or misspelled file dataset reaches a run unchecked. Take config and entity as the reference implementations here, not file.