Test-only changes¶

The test-only changes job does not run the full set of PHPUnit tests, but instead it just runs the tests that have been modified. Additionally all of the non-test files are reverted back to their unchanged state.

The purpose of this job is to demonstrate that the changes in the tests provide test coverage for the feature/bug/problem which is being worked on. It is equivalent to the "test-only will fail" patches that have been used on DrupalCI for contrib projects and for core over the past years. Drupal Core already has a 'test-only changes' GitLab pipeline job, but this feature is new for contrib projects.

This job is manual, which means it is optional and will not start automatically, but requires you to click the [>] 'play' button.

The log has extra information showing which files have been changed in the merge request and which files have been reverted before running the tests. The job is intended to fail with an amber warning, as this demonstrates that the modified tests do provide test coverage for the problem being fixed.

If this job ends with a green pass it means that the bug or feature does not have test coverage, because the tests pass even with the unmodified source files.

It can also be used to demonstrate new test coverage for existing source code which is not being changed. To do this, make a temporary change to the source (to simulate a bug) which will cause the new test to fail in the regular phpunit job. However the test-only changes job should pass green. This is the reverse outcome compared the usual scenario described above, but it does indicate that the new test provides coverage as intended.

Merge Requests¶

The most usual place where this job is run is via a Merge Request which contains changes to PHPUnit tests. The job will be added automatically if changes are detected between the MR branch and the target branch.

Manual Pipelines¶

It is also possible to trigger the test-only changes job without needing to create a merge request. This is achieved by checking for differences compared to the branch named in the _TEST_ONLY_TARGET_BRANCH variable. The default is the project's own default branch.

If you need to compare against a specific baseline then enter the commit hash in the _TEST_ONLY_TARGET_BASELINE variable. This is optional and will default to the tip of the branch named in _TEST_ONLY_TARGET_BRANCH.

variables:
  # The branch to check against for changes in test files (optional, defaults to the project's default branch)
  _TEST_ONLY_TARGET_BRANCH: '1.0.x'
  # The commit hash reference for the branch (optional)
  _TEST_ONLY_TARGET_BASELINE: 'a1b2c3d4e5f67890abcdef1234567890abcdef12'

Branch Pipelines¶

The test-only changes job can be run in a branch pipeline, using the same _TEST_ONLY_TARGET_BRANCH and _TEST_ONLY_TARGET_BASELINE as described above.

Keep specific non-test files¶

If you require changes to specific non-test files to be retained in order to successfully run the tests, this can be done by naming those files in the _TEST_ONLY_FILE_PATTERN variable. This can be a single filename, or multiple files can be matched using regex pattern matching. For example, to retain the changes to the file module.services.yml and all data{*}.dat files, then the following regex could be used:

variables:
  _TEST_ONLY_FILE_PATTERN: 'module\.services\.yml|^data*\.dat'

The variable should typically be temporarily set inside the job. This can be done in two ways:

Clicking the test-only changes pipeline and triggering a manual action with the extra variables. You can also update the variables on a previously ran job by clicking the Update CI/CD variables button on the top right corner of the page. You should see something like this:
Creating a new manual pipeline with the appropriate variables.

Parallel option¶

If the phpunit job uses parallel, those options will be inherited by the test-only changes job. The parallel options could be a number, to split a job into several ones, or could be a matrix of options, each one passed to a different parallel job. This means that in some cases, keeping this value makes sense but in others it does not.

If you want the test-only changes job to not have parallel options, you can do this in your .gitlab-ci.yml:

test-only changes:
  # Remove the parallel definition that is defined in the customized phpunit job.
  parallel:

before_script and after_script¶

The test-only changes job definition extends the phpunit definition, so if your project has added a before_script or after_script to phpunit then these will also be run by the test-only changes job. If you do not want this to happen, then remove them using:

test-only changes:
  before_script:
  after_script: