Skip to content

Writing a dataset source plugin

How to load eval questions from a store ai_eval does not ship a reader for: a remote registry, a git checkout, a third-party test-case API. For PHP developers. For the dataset file format itself, read datasets.

A dataset source is a plugin class in your module's Plugin/AiEvalDatasetSource namespace, carrying the AiEvalDatasetSource attribute and implementing DatasetSourceInterface, normally by extending DatasetSourceBase. ai_eval ships three: file, config and entity.

How a source is reached

An evaluation target names a source plugin ID and a source-relative reference. Service\DatasetLoader is a thin resolver that instantiates the named plugin and calls it. Everything about parsing, validating and listing belongs to the plugin.

flowchart LR
  T["Evaluation target<br/>source id plus ref"] --> DL["DatasetLoader"]
  DL --> PM["DatasetSourcePluginManager"]
  PM --> P["Your source plugin"]
  P --> ST[("Your store")]
  P --> Q["Question rows"]
  Q --> R["EvalRunner"]

An unknown source ID is logged and returns an empty question list. It never throws.

The attribute

Drupal\ai_eval\Attribute\AiEvalDatasetSource takes three properties.

Property Type Default Meaning
id string required The plugin ID. Targets store this string.
label string required Human-readable name, shown in source selectors.
description string '' What this source loads from.

The interface

Drupal\ai_eval\DatasetSourceInterface has four methods. DatasetSourceBase implements the last two, so a read-only source normally writes two.

Method Returns Contract
load(string $ref, ?string $questionId = NULL) array<int, array<string, mixed>> Question rows for one dataset reference, re-indexed. Disabled questions excluded. Filter to one question when $questionId is given.
listDatasets() array<int, string> The source-relative references this source can load.
listDatasetsWithLabels() array<string, string> Map of reference to human label. Base returns ref => ref.
capabilities() array<int, string> Subset of read, write, promote. Base returns ['read']. Always contains read.

load() must not throw. A reference that does not resolve, a parse failure, or a document that fails schema validation returns an empty array, and should log a warning on the ai_eval log channel so the empty result is explainable.

The shipped file source validates the whole document before normalization and logs missing files, parse failures, and schema errors. Invalid files are refused as a whole rather than evaluated with malformed questions silently dropped.

DatasetSourceBase::normalizeQuestions() does the common post-processing for you: it drops rows that are not arrays (logging each one), drops rows with a truthy disabled, applies the $questionId filter, and re-indexes. Call it with the raw rows, a label for log messages, the question ID and a logger.

Optional capability interfaces

These are separate interfaces, not methods on DatasetSourceInterface, so an existing third-party source stays valid without changes.

Interface Adds Implemented by
WritableDatasetSourceInterface appendQuestions(), createDataset() entity
SplitAwareDatasetSourceInterface loadSplitsConfig() file, config

WritableDatasetSourceInterface is the single seam that promotion, synthetic generation and any future writer funnel through. appendQuestions() validates every question against the dataset JSON Schema before persisting anything, and persists none of them if any one fails. It allocates non-colliding question IDs and returns them in input order. Detect support with instanceof WritableDatasetSourceInterface for type safety, or with capabilities() when the UI just needs to know what to offer.

SplitAwareDatasetSourceInterface exposes the dataset-level splits block, which load() cannot return because it yields question rows only. Implement it when your backing store holds the whole dataset document. Sources without document-level metadata simply do not implement it, and only explicit per-question split labels then apply. Shape validation of the block is DatasetSplitter's job, not yours: return the raw block or [].

A complete example

A read-only source that fetches dataset documents over HTTP from a registry, validates each against the shipped dataset schema, and caches the parsed result for the request.

my_eval_extras/src/Plugin/AiEvalDatasetSource/RegistrySource.php:

<?php

declare(strict_types=1);

namespace Drupal\my_eval_extras\Plugin\AiEvalDatasetSource;

use Drupal\ai_eval\Attribute\AiEvalDatasetSource;
use Drupal\ai_eval\DatasetSourceBase;
use Drupal\ai_eval\Service\DatasetValidator;
use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use GuzzleHttp\ClientInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\Yaml\Yaml;

/**
 * Loads eval datasets from a remote Eval Commons registry over HTTP.
 *
 * The ref is a bare document name; the base URL comes from
 * my_eval_extras.settings:registry_base_url. Documents are validated against
 * the ai_eval dataset JSON Schema before their questions are returned, so a
 * registry serving something else cannot poison a run.
 */
#[AiEvalDatasetSource(
  id: 'registry',
  label: 'Registry',
  description: 'Loads datasets from a remote Eval Commons registry over HTTP.',
)]
final class RegistrySource extends DatasetSourceBase implements ContainerFactoryPluginInterface {

  /**
   * Parsed documents, keyed by ref, for the life of the request.
   *
   * @var array<string, array<string, mixed>|null>
   */
  private array $cache = [];

  public function __construct(
    array $configuration,
    string $plugin_id,
    mixed $plugin_definition,
    private readonly ClientInterface $httpClient,
    private readonly ConfigFactoryInterface $configFactory,
    private readonly DatasetValidator $datasetValidator,
    private readonly LoggerInterface $channelLogger,
  ) {
    parent::__construct($configuration, $plugin_id, $plugin_definition);
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition): static {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $container->get('http_client'),
      $container->get('config.factory'),
      $container->get('ai_eval.dataset_validator'),
      $container->get('logger.channel.ai_eval'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function load(string $ref, ?string $questionId = NULL): array {
    $document = $this->fetch($ref);
    if ($document === NULL) {
      return [];
    }

    return $this->normalizeQuestions(
      $document['questions'] ?? [],
      'registry:' . $ref,
      $questionId,
      $this->channelLogger,
    );
  }

  /**
   * {@inheritdoc}
   */
  public function listDatasets(): array {
    $index = $this->configFactory
      ->get('my_eval_extras.settings')
      ->get('registry_documents');

    return is_array($index) ? array_values(array_filter($index, 'is_string')) : [];
  }

  /**
   * Fetches and validates one registry document.
   *
   * Never throws: a run must degrade to zero questions with a log line, not
   * abort because a registry was briefly unreachable.
   *
   * @param string $ref
   *   The document name.
   *
   * @return array<string, mixed>|null
   *   The validated document, or NULL when it could not be loaded.
   */
  private function fetch(string $ref): ?array {
    if (array_key_exists($ref, $this->cache)) {
      return $this->cache[$ref];
    }
    $this->cache[$ref] = NULL;

    // A ref is a bare document name. Anything path-like is refused rather
    // than sent, so a crafted target cannot reach outside the registry.
    if ($ref === '' || basename($ref) !== $ref) {
      $this->channelLogger->warning('RegistrySource: refusing non-simple dataset ref @ref.', ['@ref' => $ref]);
      return NULL;
    }

    $base = rtrim((string) $this->configFactory->get('my_eval_extras.settings')->get('registry_base_url'), '/');
    if ($base === '') {
      $this->channelLogger->warning('RegistrySource: no registry_base_url configured.');
      return NULL;
    }

    try {
      $body = (string) $this->httpClient
        ->request('GET', $base . '/' . $ref, ['timeout' => 10])
        ->getBody();
      $document = Yaml::parse($body);
    }
    catch (\Throwable $e) {
      $this->channelLogger->warning('RegistrySource: could not load @ref: @message', [
        '@ref' => $ref,
        '@message' => $e->getMessage(),
      ]);
      return NULL;
    }

    if (!is_array($document) || !$this->datasetValidator->validate($document)) {
      $this->channelLogger->warning('RegistrySource: @ref failed dataset schema validation: @errors', [
        '@ref' => $ref,
        '@errors' => implode('; ', $this->datasetValidator->errors()),
      ]);
      return NULL;
    }

    $this->cache[$ref] = $document;
    return $document;
  }

}

Clear caches and registry appears as a dataset source on the target form.

Validate what you load

All three shipped sources (file, config, and entity) validate the document with ai_eval.dataset_validator before returning questions. Invalid documents return an empty array and log the reason. Custom sources should follow the same contract.

The file source discovers .yaml and .yml datasets in this precedence order: the configured dataset_path, ai_eval's legacy data/ directory, then the datasets/ directories of enabled modules in module-name order. A duplicate filename resolves to the first directory and logs the ignored duplicate. Listing, loading questions, and loading split configuration use the same resolution rules.