Automated Functional Acceptance Testing

The functional acceptance suite uses varbase-e2e 2.0.x, a Behaviour-Driven Development layer over Playwright and Cucumber-js. Scenarios are written in plain Gherkin and driven through a real browser.

Principles

Black box. Scenarios drive the site the way an editor does: through pages, forms and the editing surface. There are no Drush calls and no shell commands inside a scenario. Anything the site needs beforehand is set up by the pipeline or the test recipe, not by the test.

Meaningful assertions. Scenarios assert the behaviour the module promises, not that a page returned 200 and not that a theme rendered a particular wrapper. The anchor markup, the toolbar button, the saved output and the access rules are what get checked.

One concern per feature. Each .feature file covers a single behaviour, so a failure names the thing that broke.

Configuration

cucumber.js at the module root configures the run:

module.exports = {
  default: {
    timeout: 60000,
    requireModule: ['tsx/cjs'],
    require: [
      'node_modules/@vardot/varbase-e2e/tests/step-definitions/**/*.js',
      'tests/step-definitions/**/*.js',
    ],
    paths: ['tests/features/drupal/**/*.feature'],
    format: [
      '@cucumber/pretty-formatter',
      'json:tests/reports/cucumber_report.json',
    ],
    worldParameters: { /* ... */ },
  },
};

Key points:

Reusable steps ship with varbase-e2e; module-specific steps live in tests/step-definitions/anchor_link.steps.js.
Only tests/features/drupal/** is loaded.
The 60 second step timeout accommodates the first CKEditor 5 boot, which is the slowest thing in the suite.

World parameters

Parameter	Purpose
`launchUrl`	Base URL, from `LAUNCH_URL`, defaulting to `http://localhost`.
`users`	Registry of test users per role.
`selectors`	Named-selector registries loaded from `tests/selectors/`.
`screenshot`	Screenshots on failure, into `tests/screenshots/`.
`video`	Video `on-failure`, into `tests/videos/`.
`javascript`	Console watching: `mode: 'warn'`, `levels: ['error']`, checked after each scenario.

That last one is worth calling out. The suite watches the browser console for errors after every scenario, so a JavaScript error introduced by the editor library surfaces as test output rather than going unnoticed.

Users

Three users, defined in cucumber.js:

Name	Username	Role
Webmaster	`webmaster`	Site install super-admin, `isAdmin: true`
Content editor	`content_editor_user`	`content_editor`
Authenticated user	`authenticated_user`	none

The Webmaster is created by the site install. The rest are provisioned by the Given I add testing users step in the first feature, which iterates the registry and skips entries flagged isAdmin.

This is why 01-01-01-users-login.feature runs first and is tagged @setup: every later feature depends on those users existing.

Named selectors

Rather than scattering CSS strings through the steps, selectors are registered by name in JSON under tests/selectors/:

anchor_link.json - module-specific selectors
cms-drupal-core-claro.json - shared Claro admin theme presets

Steps refer to a selector by name. When a Drupal or theme release moves an element, the fix is one JSON entry rather than a search across every feature file.

The test recipe

tests/recipes/anchor_link_test/ provisions the environment the suite expects:

an Anchor Test CKEditor 5 text format (editor.editor.anchor_test, filter.format.anchor_test) with the Anchor button in the toolbar;
the article content type and its body field storage and instance;
the form and view displays for node.article.

Applying the recipe is what makes the suite reproducible: the format under test is defined in configuration, not clicked together by hand.

drush recipe /path/to/anchor_link/tests/recipes/anchor_link_test -y

Reports and artefacts

A run produces, under tests/:

reports/cucumber_report.json - machine-readable results
reports/cucumber_report.html and .pdf - generated by @vardot/varbase-e2e/bin/generate-reports.js
screenshots/ - on failure, prefixed failed_
videos/ - on failure

None of these are committed.

Report generation is run explicitly rather than through varbase-e2e's automatic exit hook, because that hook cannot await PDF generation from process.on('exit'). The pipeline sets VARBASE_E2E_REPORT_DISABLE=1 and calls generate-reports.js afterwards, so the HTML and PDF land next to the JSON.