Extending AI Eval

What a module can add to AI Eval, and where each extension point plugs into a run. For PHP developers writing a module that sits alongside ai_eval.

There are four extension points, plus two plugin-definition alters. Everything here is discovered from any enabled module, so nothing below needs a patch to ai_eval itself.

Extension point	Mechanism	Contract
Grader	Plugin in Plugin/AiEvalGrader	GraderInterface
Dataset source	Plugin in Plugin/AiEvalDatasetSource	DatasetSourceInterface
Check executor	Service tagged ai_eval.check_executor	CheckExecutorInterface
Run event	Event subscriber	EvalRunCompleteEvent

Where each one sits in a run

flowchart TD
  DS["Dataset source plugin<br/>load(ref)"] --> Q["Question rows"]
  Q --> RUN["EvalRunner"]
  RUN --> INV["Invoke target<br/>agent mode or chat mode"]
  INV --> RESP["Response text<br/>plus tool-call record"]
  RESP --> GR["Grader plugins<br/>grade(input, response, context)"]
  GR --> RCG["rubric_checks grader"]
  RCG --> RCE["RubricCheckEvaluator"]
  RCE --> EX["Check executor services"]
  GR --> SC["Scorer<br/>composite score and gate"]
  SC --> ROW["ai_eval_result row"]
  ROW --> EV["EvalRunCompleteEvent"]

A grader scores a whole response on one dimension. A check executor runs one declarative check inside a rubric, and reaches the run only through the rubric_checks grader, which resolves the question's rubric_ref and combines the per-check outcomes into a single grade.

Plugin discovery

Both plugin types use a DefaultPluginManager subclass with attribute discovery. Put the class in your module's Plugin/AiEvalGrader or Plugin/AiEvalDatasetSource namespace, add the attribute, and clear caches.

Manager service	Namespace scanned	Attribute
ai_eval.grader_manager	Plugin/AiEvalGrader	Drupal\ai_eval\Attribute\AiEvalGrader
ai_eval.dataset_source_manager	Plugin/AiEvalDatasetSource	Drupal\ai_eval\Attribute\AiEvalDatasetSource

Check executors are ordinary services collected on a tag, so they are declared in your *.services.yml and need no attribute.

Altering definitions you do not own

The two managers each raise an alter hook, documented with example implementations in ai_eval.api.php at the module root:

• hook_ai_eval_grader_info_alter(array &$definitions)
• hook_ai_eval_dataset_source_info_alter(array &$definitions)

These are the only two hooks ai_eval invokes.

Services worth knowing

Injectable services a plugin or subscriber commonly needs.

Service ID	Class	What it does
ai_eval.grader_manager	GraderPluginManager	Grader discovery, plus actionObservingGraders()
ai_eval.dataset_source_manager	DatasetSourcePluginManager	Dataset source discovery
ai_eval.dataset_loader	Service\DatasetLoader	Resolves a target's dataset through its source
ai_eval.dataset_validator	Service\DatasetValidator	Validates a dataset document against dataset.schema.json
ai_eval.rubric_validator	Service\RubricValidator	Validates a rubric, including executor-provided check kinds
ai_eval.rubric_loader	Service\RubricLoader	Resolves a rubric_ref to a rubric document
ai_eval.rubric_check_evaluator	Service\RubricCheckEvaluator	Dispatches checks to executors and combines outcomes
ai_eval.check_template_resolver	Service\CheckTemplateResolver	Resolves metadata. and expected. dot-paths from a question
ai_eval.grader_scale_resolver	Service\GraderScaleResolver	Reads a grader's declared score scale from its definition
ai_eval.rate_limit_handler	Service\RateLimitHandler	Throttle and retry around provider calls
ai_eval.judge_config_store	Service\JudgeConfigStore	Stored per-judge prompt overrides
logger.channel.ai_eval	Logger channel	The module's log channel

Value objects have no service: GradeResult, CheckOutcome and ScoreScale are plain classes in the Drupal\ai_eval namespace.

The portable schemas

Dataset, rubric and judge YAML are described by public JSON Schemas that ai_eval ships and implements. If you are writing a tool that reads or writes eval content rather than a Drupal plugin, start at the schemas.